2024-03-25T16:23:08.672 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:23:08.673 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278/orig.config.yaml 2024-03-25T16:23:08.717 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:23:08.800 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621278' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi099.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGtxbdLy2UHaHA5gRIf3MR5NU+UzyKuyLZnQy02lhjxuvfdtMPt2OxqNQS24ovTq2LYanLIoZBf0ViH2OnEuAKo= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCYd91K16MqN2SbtpXlvonTKb/ApH9+7u27YGHa817euhxeRbawaNy+ucBzzgJtxkgmTJjO1E+7tv7Dyw3xr/rU= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:23:08.800 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:23:08.801 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:23:08.801 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:23:08.802 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:23:08.826 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:23:08.826 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:23:08.827 INFO:teuthology.packaging:ref: None 2024-03-25T16:23:08.827 INFO:teuthology.packaging:tag: None 2024-03-25T16:23:08.827 INFO:teuthology.packaging:branch: main 2024-03-25T16:23:08.827 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:23:08.827 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:23:09.152 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:23:09.153 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:23:09.160 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:23:09.160 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:23:09.186 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:23:09.203 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:23:09.210 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:23:09.232 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:18:06.649998', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGtxbdLy2UHaHA5gRIf3MR5NU+UzyKuyLZnQy02lhjxuvfdtMPt2OxqNQS24ovTq2LYanLIoZBf0ViH2OnEuAKo='} 2024-03-25T16:23:09.255 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:18:06.648321', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCYd91K16MqN2SbtpXlvonTKb/ApH9+7u27YGHa817euhxeRbawaNy+ucBzzgJtxkgmTJjO1E+7tv7Dyw3xr/rU='} 2024-03-25T16:23:09.256 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:23:09.262 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:23:09.262 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:23:09.262 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:23:09.354 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb917b720d0>, signals=[15]) 2024-03-25T16:23:09.355 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:23:09.364 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:23:09.364 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2024-03-25T16:23:09.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:23:09.459 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-03-25T16:23:09.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:23:09.548 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:23:09.555 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2024-03-25T16:23:09.573 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2024-03-25T16:23:09.573 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2024-03-25T16:23:09.630 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="CentOS Stream" 2024-03-25T16:23:09.630 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="9" 2024-03-25T16:23:09.630 INFO:teuthology.orchestra.run.smithi099.stdout:ID="centos" 2024-03-25T16:23:09.630 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:23:09.630 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="9" 2024-03-25T16:23:09.630 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:23:09.630 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:23:09.630 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2024-03-25T16:23:09.630 INFO:teuthology.orchestra.run.smithi099.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:23:09.630 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:23:09.630 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:23:09.630 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:23:09.631 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:23:09.631 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:23:09.631 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2024-03-25T16:23:09.657 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-03-25T16:23:09.675 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-03-25T16:23:09.676 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-03-25T16:23:09.733 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2024-03-25T16:23:09.733 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="9" 2024-03-25T16:23:09.733 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2024-03-25T16:23:09.734 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:23:09.734 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="9" 2024-03-25T16:23:09.734 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:23:09.734 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:23:09.734 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2024-03-25T16:23:09.734 INFO:teuthology.orchestra.run.smithi190.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:23:09.734 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:23:09.734 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:23:09.734 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:23:09.734 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:23:09.734 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:23:09.734 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-03-25T16:23:09.757 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:23:09.766 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:23:09.785 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:23:09.786 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:23:09.788 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:23:09.804 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:23:09.810 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:23:09.810 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:23:09.845 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:23:09.922 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:23:09.969 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:23:09.983 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:23:09.983 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:23:09.984 DEBUG:teuthology.orchestra.run.smithi099:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:23:09.984 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:23:10.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:23:10.000 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-03-25T16:23:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:23:10.001 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-03-25T16:23:10.058 INFO:teuthology.orchestra.run.smithi099.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:23:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:23:10.059 INFO:teuthology.task.kernel:Running kernel on smithi099: 5.14.0-361.el9.x86_64 2024-03-25T16:23:10.059 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum install -y kernel 2024-03-25T16:23:10.059 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.14.0-361.el9.x86_64 2024-03-25T16:23:10.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum install -y kernel 2024-03-25T16:23:12.104 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T16:23:12.246 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T16:23:13.730 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-25T16:23:13.871 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-25T16:23:14.820 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-25T16:23:15.111 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - BaseOS 8.8 MB/s | 8.0 MB 00:00 2024-03-25T16:23:16.756 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-25T16:23:18.096 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 19 MB 00:01 2024-03-25T16:23:21.071 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 7.2 MB/s | 6.1 MB 00:00 2024-03-25T16:23:22.555 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-25T16:23:23.487 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.1 MB 00:01 2024-03-25T16:23:25.146 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-03-25T16:23:25.940 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 6.3 MB/s | 21 MB 00:03 2024-03-25T16:23:31.006 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-25T16:23:33.097 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:23:33.097 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:23:33.177 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:23:33.185 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:23:33.185 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-25T16:23:33.185 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:23:33.185 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:23:33.185 INFO:teuthology.orchestra.run.smithi190.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:23:33.186 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-25T16:23:33.186 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:23:33.186 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:23:33.186 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:23:33.186 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:23:33.186 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:23:33.186 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:23:33.187 INFO:teuthology.orchestra.run.smithi190.stdout:Install 4 Packages 2024-03-25T16:23:33.187 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:23:33.187 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 76 M 2024-03-25T16:23:33.188 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 123 M 2024-03-25T16:23:33.188 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:23:33.325 INFO:teuthology.orchestra.run.smithi190.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 491 kB/s | 44 kB 00:00 2024-03-25T16:23:33.666 INFO:teuthology.orchestra.run.smithi190.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 34 MB/s | 15 MB 00:00 2024-03-25T16:23:34.117 INFO:teuthology.orchestra.run.smithi190.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 38 MB/s | 33 MB 00:00 2024-03-25T16:23:34.367 INFO:teuthology.orchestra.run.smithi190.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 26 MB/s | 27 MB 00:01 2024-03-25T16:23:34.367 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:34.367 INFO:teuthology.orchestra.run.smithi190.stdout:Total 64 MB/s | 76 MB 00:01 2024-03-25T16:23:35.027 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:23:35.390 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:23:35.390 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:23:36.387 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:23:36.387 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:23:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:23:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:23:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:23:42.327 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:23:42.470 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:23:47.187 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:23:47.228 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:23:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:24:21.899 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:24:21.899 INFO:teuthology.orchestra.run.smithi190.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:24:21.899 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:24:21.949 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:24:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:24:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:24:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:24:22.352 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:24:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:24:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:24:22.787 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:24:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:24:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:24:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:24:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:24:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:24:22.787 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:24:23.137 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:24:23.236 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:24:23.236 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:24:23.236 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:24:23.237 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:24:23.237 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:24:23.237 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:24:23.237 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:24:23.237 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:24:23.317 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:24:23.317 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:24:23.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo rpm -qi grub2-tools 2024-03-25T16:24:23.351 INFO:teuthology.orchestra.run.smithi190.stdout:Name : grub2-tools 2024-03-25T16:24:23.351 INFO:teuthology.orchestra.run.smithi190.stdout:Epoch : 1 2024-03-25T16:24:23.351 INFO:teuthology.orchestra.run.smithi190.stdout:Version : 2.06 2024-03-25T16:24:23.351 INFO:teuthology.orchestra.run.smithi190.stdout:Release : 68.el9 2024-03-25T16:24:23.351 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: x86_64 2024-03-25T16:24:23.351 INFO:teuthology.orchestra.run.smithi190.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:24:23.351 INFO:teuthology.orchestra.run.smithi190.stdout:Group : Unspecified 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout:Size : 8264254 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout:License : GPLv3+ 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout:Packager : builder@centos.org 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout:Vendor : CentOS 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout:Summary : Support tools for GRUB. 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout:Description : 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:24:23.352 INFO:teuthology.orchestra.run.smithi190.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:24:23.353 INFO:teuthology.orchestra.run.smithi190.stdout:hardware devices. 2024-03-25T16:24:23.353 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:24:23.353 INFO:teuthology.orchestra.run.smithi190.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:24:23.353 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:24:23.354 INFO:teuthology.task.kernel:Updating grub on smithi190 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:24:23.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:24:23.983 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2024-03-25T16:24:25.370 INFO:teuthology.orchestra.run.smithi190.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:24:25.405 INFO:teuthology.orchestra.run.smithi190.stderr:done 2024-03-25T16:24:25.408 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-03-25T16:24:25.423 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.gVqs9RbCxg 2024-03-25T16:24:25.699 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gVqs9RbCxg 2024-03-25T16:24:25.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.gVqs9RbCxg 2024-03-25T16:24:25.893 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.gVqs9RbCxg is 6KB 2024-03-25T16:24:25.940 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.gVqs9RbCxg 2024-03-25T16:24:25.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:24:26.020 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:24:26.020 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:24:26.020 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:24:26.020 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:24:26.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:24:26.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2024-03-25T16:24:56.178 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:24:56.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-25T16:24:56.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:25:14.530 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-03-25T16:25:23.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-25T16:25:23.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:25:26.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-03-25T16:25:38.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-25T16:25:38.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:25:40.045 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 157 kB/s | 21 MB 02:14 2024-03-25T16:25:44.936 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-25T16:25:47.019 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:25:47.019 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:25:47.101 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:25:47.109 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:25:47.109 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-03-25T16:25:47.109 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:25:47.109 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:25:47.109 INFO:teuthology.orchestra.run.smithi099.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:25:47.109 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-03-25T16:25:47.109 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:25:47.109 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:25:47.109 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:25:47.109 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:25:47.109 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:25:47.109 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:25:47.110 INFO:teuthology.orchestra.run.smithi099.stdout:Install 4 Packages 2024-03-25T16:25:47.110 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:25:47.110 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 76 M 2024-03-25T16:25:47.110 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 123 M 2024-03-25T16:25:47.110 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:25:47.338 INFO:teuthology.orchestra.run.smithi099.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 269 kB/s | 44 kB 00:00 2024-03-25T16:25:48.338 INFO:teuthology.orchestra.run.smithi099.stdout:(2/4): kernel-modules-core-5.14.0-430.el9.x86_6 27 MB/s | 27 MB 00:00 2024-03-25T16:25:48.521 INFO:teuthology.orchestra.run.smithi099.stdout:(3/4): kernel-core-5.14.0-430.el9.x86_64.rpm 11 MB/s | 15 MB 00:01 2024-03-25T16:25:49.088 INFO:teuthology.orchestra.run.smithi099.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 17 MB/s | 33 MB 00:01 2024-03-25T16:25:49.089 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:49.089 INFO:teuthology.orchestra.run.smithi099.stdout:Total 38 MB/s | 76 MB 00:01 2024-03-25T16:25:49.752 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:25:50.122 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:25:50.122 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:25:51.130 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:25:51.131 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:25:53.599 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:25:54.192 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:25:54.272 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:25:57.033 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:25:57.177 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:26:01.856 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:26:01.968 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:06.690 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:10.498 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-03-25T16:26:10.964 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-25T16:26:10.964 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:26:10.965 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-03-25T16:26:10.979 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:26:10.979 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:26:10.979 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:26:10.979 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:26:11.980 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:26:11.980 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:26:12.067 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-03-25T16:26:12.107 DEBUG:teuthology.parallel:result is None 2024-03-25T16:26:35.876 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:35.877 INFO:teuthology.orchestra.run.smithi099.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:26:35.877 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:26:35.920 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:36.284 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:36.284 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:26:36.284 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:26:36.284 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:26:36.660 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:36.660 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:26:36.660 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:26:36.660 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:36.660 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:26:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:26:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:26:36.661 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:26:36.661 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:26:36.983 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:26:37.092 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:37.092 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:26:37.092 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:26:37.092 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:26:37.093 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:26:37.093 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:26:37.093 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi099.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:26:37.093 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:26:37.173 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:37.173 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:26:37.173 DEBUG:teuthology.orchestra.run.smithi099:> sudo rpm -qi grub2-tools 2024-03-25T16:26:37.215 INFO:teuthology.orchestra.run.smithi099.stdout:Name : grub2-tools 2024-03-25T16:26:37.215 INFO:teuthology.orchestra.run.smithi099.stdout:Epoch : 1 2024-03-25T16:26:37.215 INFO:teuthology.orchestra.run.smithi099.stdout:Version : 2.06 2024-03-25T16:26:37.215 INFO:teuthology.orchestra.run.smithi099.stdout:Release : 68.el9 2024-03-25T16:26:37.215 INFO:teuthology.orchestra.run.smithi099.stdout:Architecture: x86_64 2024-03-25T16:26:37.215 INFO:teuthology.orchestra.run.smithi099.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:26:37.215 INFO:teuthology.orchestra.run.smithi099.stdout:Group : Unspecified 2024-03-25T16:26:37.215 INFO:teuthology.orchestra.run.smithi099.stdout:Size : 8264254 2024-03-25T16:26:37.216 INFO:teuthology.orchestra.run.smithi099.stdout:License : GPLv3+ 2024-03-25T16:26:37.216 INFO:teuthology.orchestra.run.smithi099.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:26:37.216 INFO:teuthology.orchestra.run.smithi099.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:26:37.216 INFO:teuthology.orchestra.run.smithi099.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:26:37.216 INFO:teuthology.orchestra.run.smithi099.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:26:37.216 INFO:teuthology.orchestra.run.smithi099.stdout:Packager : builder@centos.org 2024-03-25T16:26:37.216 INFO:teuthology.orchestra.run.smithi099.stdout:Vendor : CentOS 2024-03-25T16:26:37.216 INFO:teuthology.orchestra.run.smithi099.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:26:37.216 INFO:teuthology.orchestra.run.smithi099.stdout:Summary : Support tools for GRUB. 2024-03-25T16:26:37.216 INFO:teuthology.orchestra.run.smithi099.stdout:Description : 2024-03-25T16:26:37.216 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:26:37.216 INFO:teuthology.orchestra.run.smithi099.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:26:37.216 INFO:teuthology.orchestra.run.smithi099.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:26:37.216 INFO:teuthology.orchestra.run.smithi099.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:26:37.217 INFO:teuthology.orchestra.run.smithi099.stdout:hardware devices. 2024-03-25T16:26:37.217 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:26:37.217 INFO:teuthology.orchestra.run.smithi099.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:26:37.218 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:26:37.218 INFO:teuthology.task.kernel:Updating grub on smithi099 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:26:37.218 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:26:37.835 INFO:teuthology.orchestra.run.smithi099.stderr:Generating grub configuration file ... 2024-03-25T16:26:39.200 INFO:teuthology.orchestra.run.smithi099.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:26:39.256 INFO:teuthology.orchestra.run.smithi099.stderr:done 2024-03-25T16:26:39.259 DEBUG:teuthology.orchestra.run.smithi099:> mktemp 2024-03-25T16:26:39.274 INFO:teuthology.orchestra.run.smithi099.stdout:/tmp/tmp.ab6VrCHlsE 2024-03-25T16:26:39.274 DEBUG:teuthology.orchestra.run.smithi099:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ab6VrCHlsE 2024-03-25T16:26:39.344 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0666 /tmp/tmp.ab6VrCHlsE 2024-03-25T16:26:39.479 DEBUG:teuthology.orchestra.remote:smithi099:/tmp/tmp.ab6VrCHlsE is 6KB 2024-03-25T16:26:39.527 DEBUG:teuthology.orchestra.run.smithi099:> rm -fr /tmp/tmp.ab6VrCHlsE 2024-03-25T16:26:39.542 DEBUG:teuthology.orchestra.run.smithi099:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:26:39.606 INFO:teuthology.orchestra.run.smithi099.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:26:39.606 INFO:teuthology.orchestra.run.smithi099.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:26:39.606 INFO:teuthology.orchestra.run.smithi099.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:26:39.606 INFO:teuthology.orchestra.run.smithi099.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:26:39.608 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:26:39.774 DEBUG:teuthology.orchestra.run.smithi099:> sudo shutdown -r now 2024-03-25T16:27:09.777 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:27:09.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-25T16:27:09.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:27:28.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.99 2024-03-25T16:27:37.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-25T16:27:37.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:27:40.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.99 2024-03-25T16:27:52.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-25T16:27:52.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:28:10.659 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.99 2024-03-25T16:28:25.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-25T16:28:25.661 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:28:26.017 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-03-25T16:28:26.462 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-25T16:28:26.462 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:28:26.462 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-03-25T16:28:26.476 INFO:teuthology.orchestra.run.smithi099.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:28:26.477 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:28:26.477 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:28:26.477 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:28:27.478 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:28:27.478 DEBUG:teuthology.orchestra.run.smithi099:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:28:27.538 INFO:teuthology.orchestra.run.smithi099.stdout:ttyS1 2024-03-25T16:28:27.571 DEBUG:teuthology.parallel:result is None 2024-03-25T16:28:27.571 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:28:27.579 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:28:27.580 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:28:27.582 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:28:27.604 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:28:27.611 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:28:27.656 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:28:27.656 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:28:27.658 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:28:27.690 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:28:27.706 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:28:27.707 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:28:27.711 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:28:27.739 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:28:27.752 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:28:27.774 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:28:27.789 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:28:27.790 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:28:27.797 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:28:27.798 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:28:27.800 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:28:27.829 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:28:27.861 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:28:27.861 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:28:27.866 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:28:27.885 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:28:27.958 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:28:28.005 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:28:28.084 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:28:28.122 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T16:28:28.123 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:28:28.186 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:28:28.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:28:28.282 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:28:28.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:28:28.408 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T16:28:28.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:28:28.475 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2024-03-25T16:28:28.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-03-25T16:28:28.543 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:28:28.595 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:28:28.947 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:28:28.955 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:28:28.955 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:28:29.001 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:28:29.051 DEBUG:teuthology.orchestra.run.smithi099:> sudo service auditd rotate 2024-03-25T16:28:29.121 INFO:teuthology.orchestra.run.smithi099.stdout:Rotating logs: 2024-03-25T16:28:29.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2024-03-25T16:28:29.171 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2024-03-25T16:28:29.172 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:28:29.173 DEBUG:teuthology.orchestra.run.smithi099:> /usr/sbin/getenforce 2024-03-25T16:28:29.197 INFO:teuthology.orchestra.run.smithi099.stdout:Permissive 2024-03-25T16:28:29.198 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2024-03-25T16:28:29.223 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2024-03-25T16:28:29.223 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi099.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:28:29.224 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:28:29.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:28:29.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:28:29.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:28:29.292 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:28:29.293 DEBUG:teuthology.orchestra.run.smithi099:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:28:29.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:28:29.375 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:28:29.385 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:28:29.402 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:28:29.404 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi099.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:32:41.703 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi099.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-03-25T16:32:41.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-25T16:32:41.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:32:41.797 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-03-25T16:32:41.876 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-25T16:32:41.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-25T16:32:41.877 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:32:41.964 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-03-25T16:32:42.046 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-25T16:32:42.047 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:32:42.057 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:32:42.057 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:32:42.058 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:32:42.060 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:32:42.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:32:42.089 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:32:42.106 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:32:42.117 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:32:42.134 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:32:42.135 INFO:teuthology.orchestra.run.smithi099.stderr:sudo: ntpd: command not found 2024-03-25T16:32:42.148 INFO:teuthology.orchestra.run.smithi099.stdout:506 Cannot talk to daemon 2024-03-25T16:32:42.163 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2024-03-25T16:32:42.165 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:32:42.177 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2024-03-25T16:32:42.181 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:32:42.193 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:32:42.210 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:32:42.241 INFO:teuthology.orchestra.run.smithi099.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:32:42.244 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:32:42.244 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-03-25T16:32:42.245 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:42.245 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:42.245 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:42.245 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:42.262 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:32:42.265 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:32:42.266 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-03-25T16:32:42.266 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:42.266 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:42.266 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:42.266 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:42.266 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:32:42.275 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:32:42.275 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:32:42.275 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:32:42.275 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:32:42.281 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:32:42.281 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:32:42.281 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:32:42.282 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:32:42.284 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:32:42.284 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:32:42.511 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:32:42.512 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:32:42.524 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:32:42.524 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:32:42.854 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:32:42.855 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T16:32:42.855 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:32:42.885 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:32:42.885 DEBUG:teuthology.orchestra.run.smithi099:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:32:42.950 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:32:42.950 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T16:32:42.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:32:42.954 DEBUG:teuthology.orchestra.run.smithi099:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:32:42.979 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:32:42.980 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:32:43.035 DEBUG:teuthology.orchestra.run.smithi099:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:32:43.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:32:43.101 INFO:teuthology.orchestra.run.smithi099.stdout:check_obsoletes = 1 2024-03-25T16:32:43.103 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum clean all 2024-03-25T16:32:43.129 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:32:43.197 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2024-03-25T16:32:43.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2024-03-25T16:32:43.340 INFO:teuthology.orchestra.run.smithi099.stdout:74 files removed 2024-03-25T16:32:43.366 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-radosgw 2024-03-25T16:32:43.435 INFO:teuthology.orchestra.run.smithi190.stdout:74 files removed 2024-03-25T16:32:43.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw 2024-03-25T16:32:44.528 INFO:teuthology.orchestra.run.smithi099.stdout:ceph packages for x86_64 95 kB/s | 82 kB 00:00 2024-03-25T16:32:44.696 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 87 kB/s | 82 kB 00:00 2024-03-25T16:32:45.362 INFO:teuthology.orchestra.run.smithi099.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:32:45.594 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:32:46.229 INFO:teuthology.orchestra.run.smithi099.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T16:32:46.464 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T16:32:46.639 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-25T16:32:46.891 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-25T16:32:48.048 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-25T16:32:48.366 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-25T16:32:49.797 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.0 MB 00:01 2024-03-25T16:32:49.998 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 6.1 MB/s | 8.0 MB 00:01 2024-03-25T16:32:51.548 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-25T16:32:52.783 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-25T16:32:55.560 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-25T16:32:57.075 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 6.1 MB 00:00 2024-03-25T16:32:57.192 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-25T16:32:58.270 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-25T16:32:58.443 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-03-25T16:32:59.686 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-25T16:33:03.134 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-03-25T16:33:04.611 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-25T16:33:05.635 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:33:05.644 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:33:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-03-25T16:33:05.644 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:33:05.644 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:33:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:33:05.644 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrading: 2024-03-25T16:33:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:33:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:33:05.644 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-03-25T16:33:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: boost-program-options 2024-03-25T16:33:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:33:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:33:05.644 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:33:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:33:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:33:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:33:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:33:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:33:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:33:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:33:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:33:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:33:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:33:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:33:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:33:05.645 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:33:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:33:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse 2024-03-25T16:33:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:33:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:33:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:33:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:33:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:33:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:33:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:33:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:33:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:33:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:33:05.646 INFO:teuthology.orchestra.run.smithi099.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:33:05.647 INFO:teuthology.orchestra.run.smithi099.stdout:Installing weak dependencies: 2024-03-25T16:33:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:33:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:33:05.647 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:33:05.647 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:33:05.647 INFO:teuthology.orchestra.run.smithi099.stdout:Install 30 Packages 2024-03-25T16:33:05.647 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrade 2 Packages 2024-03-25T16:33:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:33:05.647 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 58 M 2024-03-25T16:33:05.648 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:33:07.064 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:33:07.068 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:33:07.068 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-25T16:33:07.068 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:33:07.068 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:33:07.068 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:33:07.068 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2024-03-25T16:33:07.068 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:33:07.068 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:33:07.068 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-25T16:33:07.068 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options 2024-03-25T16:33:07.068 INFO:teuthology.orchestra.run.smithi190.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:33:07.068 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:33:07.068 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:33:07.069 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:33:07.069 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:33:07.069 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:33:07.069 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:33:07.069 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:33:07.069 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:33:07.069 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:33:07.069 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:33:07.069 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:33:07.069 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:33:07.069 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:33:07.069 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:33:07.069 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:33:07.069 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:33:07.070 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse 2024-03-25T16:33:07.070 INFO:teuthology.orchestra.run.smithi190.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:33:07.070 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:33:07.070 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:33:07.070 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:33:07.070 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:33:07.070 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:33:07.070 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:33:07.070 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:33:07.070 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:33:07.070 INFO:teuthology.orchestra.run.smithi190.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:33:07.070 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:33:07.070 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2024-03-25T16:33:07.070 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:33:07.071 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:33:07.071 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:33:07.071 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:33:07.071 INFO:teuthology.orchestra.run.smithi190.stdout:Install 30 Packages 2024-03-25T16:33:07.071 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 2 Packages 2024-03-25T16:33:07.071 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:33:07.071 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 58 M 2024-03-25T16:33:07.071 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:33:23.432 INFO:teuthology.orchestra.run.smithi099.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 312 kB/s | 5.3 MB 00:17 2024-03-25T16:33:23.634 INFO:teuthology.orchestra.run.smithi099.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 125 kB/s | 25 kB 00:00 2024-03-25T16:33:26.107 INFO:teuthology.orchestra.run.smithi099.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 300 kB/s | 742 kB 00:02 2024-03-25T16:33:28.132 INFO:teuthology.orchestra.run.smithi099.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 246 kB/s | 498 kB 00:02 2024-03-25T16:33:29.368 INFO:teuthology.orchestra.run.smithi190.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 249 kB/s | 5.3 MB 00:21 2024-03-25T16:33:29.492 INFO:teuthology.orchestra.run.smithi190.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 202 kB/s | 25 kB 00:00 2024-03-25T16:33:32.251 INFO:teuthology.orchestra.run.smithi190.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 269 kB/s | 742 kB 00:02 2024-03-25T16:33:34.368 INFO:teuthology.orchestra.run.smithi190.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 235 kB/s | 498 kB 00:02 2024-03-25T16:33:39.114 INFO:teuthology.orchestra.run.smithi099.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 300 kB/s | 9.7 MB 00:33 2024-03-25T16:33:39.456 INFO:teuthology.orchestra.run.smithi099.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 132 kB/s | 45 kB 00:00 2024-03-25T16:33:40.831 INFO:teuthology.orchestra.run.smithi099.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 95 kB/s | 131 kB 00:01 2024-03-25T16:33:41.292 INFO:teuthology.orchestra.run.smithi190.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 294 kB/s | 9.7 MB 00:33 2024-03-25T16:33:41.709 INFO:teuthology.orchestra.run.smithi190.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 108 kB/s | 45 kB 00:00 2024-03-25T16:33:42.122 INFO:teuthology.orchestra.run.smithi099.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 125 kB/s | 162 kB 00:01 2024-03-25T16:33:42.609 INFO:teuthology.orchestra.run.smithi190.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 146 kB/s | 131 kB 00:00 2024-03-25T16:33:43.285 INFO:teuthology.orchestra.run.smithi190.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 240 kB/s | 162 kB 00:00 2024-03-25T16:33:44.251 INFO:teuthology.orchestra.run.smithi190.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 332 kB/s | 320 kB 00:00 2024-03-25T16:33:44.264 INFO:teuthology.orchestra.run.smithi099.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 150 kB/s | 320 kB 00:02 2024-03-25T16:33:45.039 INFO:teuthology.orchestra.run.smithi099.stdout:(10/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 309 kB/s | 5.1 MB 00:16 2024-03-25T16:33:45.218 INFO:teuthology.orchestra.run.smithi190.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 308 kB/s | 298 kB 00:00 2024-03-25T16:33:45.439 INFO:teuthology.orchestra.run.smithi099.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 253 kB/s | 298 kB 00:01 2024-03-25T16:33:45.465 INFO:teuthology.orchestra.run.smithi099.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 234 kB/s | 99 kB 00:00 2024-03-25T16:33:45.552 INFO:teuthology.orchestra.run.smithi190.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 298 kB/s | 99 kB 00:00 2024-03-25T16:33:45.607 INFO:teuthology.orchestra.run.smithi099.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 634 kB/s | 106 kB 00:00 2024-03-25T16:33:45.623 INFO:teuthology.orchestra.run.smithi099.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-25T16:33:45.648 INFO:teuthology.orchestra.run.smithi099.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-25T16:33:45.668 INFO:teuthology.orchestra.run.smithi190.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 909 kB/s | 106 kB 00:00 2024-03-25T16:33:45.716 INFO:teuthology.orchestra.run.smithi099.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-25T16:33:45.741 INFO:teuthology.orchestra.run.smithi099.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-25T16:33:45.744 INFO:teuthology.orchestra.run.smithi190.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-25T16:33:45.758 INFO:teuthology.orchestra.run.smithi099.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 996 kB/s | 41 kB 00:00 2024-03-25T16:33:45.786 INFO:teuthology.orchestra.run.smithi190.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T16:33:45.833 INFO:teuthology.orchestra.run.smithi099.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 466 kB/s | 35 kB 00:00 2024-03-25T16:33:45.867 INFO:teuthology.orchestra.run.smithi099.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:33:45.900 INFO:teuthology.orchestra.run.smithi099.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.3 MB/s | 216 kB 00:00 2024-03-25T16:33:45.928 INFO:teuthology.orchestra.run.smithi190.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 661 kB 00:00 2024-03-25T16:33:46.036 INFO:teuthology.orchestra.run.smithi190.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 282 kB 00:00 2024-03-25T16:33:46.103 INFO:teuthology.orchestra.run.smithi190.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 619 kB/s | 41 kB 00:00 2024-03-25T16:33:46.367 INFO:teuthology.orchestra.run.smithi099.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 618 kB/s | 309 kB 00:00 2024-03-25T16:33:46.442 INFO:teuthology.orchestra.run.smithi099.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-03-25T16:33:46.504 INFO:teuthology.orchestra.run.smithi190.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 541 kB/s | 216 kB 00:00 2024-03-25T16:33:46.596 INFO:teuthology.orchestra.run.smithi190.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 380 kB/s | 35 kB 00:00 2024-03-25T16:33:46.885 INFO:teuthology.orchestra.run.smithi099.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 110 kB/s | 49 kB 00:00 2024-03-25T16:33:46.887 INFO:teuthology.orchestra.run.smithi190.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 153 kB/s | 44 kB 00:00 2024-03-25T16:33:46.968 INFO:teuthology.orchestra.run.smithi099.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 806 kB/s | 67 kB 00:00 2024-03-25T16:33:47.211 INFO:teuthology.orchestra.run.smithi099.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 838 kB 00:00 2024-03-25T16:33:47.286 INFO:teuthology.orchestra.run.smithi099.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-25T16:33:47.536 INFO:teuthology.orchestra.run.smithi099.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 4.4 MB 00:01 2024-03-25T16:33:47.563 INFO:teuthology.orchestra.run.smithi190.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 457 kB/s | 309 kB 00:00 2024-03-25T16:33:47.678 INFO:teuthology.orchestra.run.smithi099.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2024-03-25T16:33:48.046 INFO:teuthology.orchestra.run.smithi190.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-03-25T16:33:48.146 INFO:teuthology.orchestra.run.smithi190.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-25T16:33:48.247 INFO:teuthology.orchestra.run.smithi190.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-03-25T16:33:48.339 INFO:teuthology.orchestra.run.smithi190.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 736 kB/s | 67 kB 00:00 2024-03-25T16:33:48.456 INFO:teuthology.orchestra.run.smithi190.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-25T16:33:48.556 INFO:teuthology.orchestra.run.smithi190.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-25T16:33:48.689 INFO:teuthology.orchestra.run.smithi190.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-25T16:33:55.414 INFO:teuthology.orchestra.run.smithi190.stdout:(29/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 424 kB/s | 20 MB 00:47 2024-03-25T16:33:56.702 INFO:teuthology.orchestra.run.smithi099.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 377 kB/s | 3.4 MB 00:09 2024-03-25T16:33:57.285 INFO:teuthology.orchestra.run.smithi099.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 333 kB/s | 3.1 MB 00:09 2024-03-25T16:33:58.572 INFO:teuthology.orchestra.run.smithi190.stdout:(30/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 216 kB/s | 5.1 MB 00:24 2024-03-25T16:34:00.689 INFO:teuthology.orchestra.run.smithi190.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 288 kB/s | 3.4 MB 00:11 2024-03-25T16:34:05.297 INFO:teuthology.orchestra.run.smithi190.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 324 kB/s | 3.1 MB 00:09 2024-03-25T16:34:05.300 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:05.300 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.0 MB/s | 58 MB 00:58 2024-03-25T16:34:05.333 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:34:05.371 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:34:05.371 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:34:05.848 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:34:05.848 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:34:07.310 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:34:07.401 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:34:07.434 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:34:07.460 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:34:07.491 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:34:07.507 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:34:07.715 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:34:07.771 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:34:08.091 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:34:08.144 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:08.236 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:08.322 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:34:08.346 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:34:08.392 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:34:08.421 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:34:08.464 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:34:08.532 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:34:08.556 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:34:08.602 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:34:08.630 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:34:08.737 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:34:08.773 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:34:08.813 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:34:08.845 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:34:08.874 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:34:08.907 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:34:09.224 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:34:09.245 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:34:09.326 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:34:09.366 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:34:09.411 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:34:09.858 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:34:09.889 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:34:09.944 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:34:10.012 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:34:10.041 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:34:12.244 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:12.279 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:12.836 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:12.874 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:34:12.968 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:34:12.968 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:34:12.969 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:34:12.969 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:34:13.009 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:22.272 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:22.272 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2024-03-25T16:34:22.272 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2024-03-25T16:34:22.272 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2024-03-25T16:34:22.272 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2024-03-25T16:34:22.273 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2024-03-25T16:34:22.273 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /root 2024-03-25T16:34:22.273 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2024-03-25T16:34:22.273 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:34:23.141 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:23.165 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:23.165 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:23.166 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:34:23.166 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:23.166 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:23.166 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:34:23.167 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:23.252 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:23.252 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:23.897 INFO:teuthology.orchestra.run.smithi099.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 261 kB/s | 20 MB 01:17 2024-03-25T16:34:23.902 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:23.902 INFO:teuthology.orchestra.run.smithi099.stdout:Total 757 kB/s | 58 MB 01:18 2024-03-25T16:34:23.934 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:34:23.973 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:34:23.973 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:34:24.343 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:24.344 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:34:24.344 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:34:24.344 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:34:24.344 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:34:24.344 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:34:24.344 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:34:24.344 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:24.344 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:34:24.344 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:34:24.344 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:34:24.344 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:34:24.345 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:34:24.345 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:34:24.345 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:34:24.345 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:34:24.345 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:34:24.345 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:34:24.345 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:34:24.345 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:34:24.345 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:34:24.345 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:34:24.345 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:34:24.345 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:34:24.345 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:34:24.345 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:34:24.346 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:34:24.346 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:34:24.346 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:34:24.346 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:34:24.346 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:34:24.346 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:24.346 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:34:24.346 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:34:24.445 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:34:24.445 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:34:25.374 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:25.375 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:34:25.375 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2024-03-25T16:34:25.375 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.375 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.375 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:34:25.375 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:34:25.375 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.375 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.375 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.375 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.375 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:34:25.375 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:34:25.376 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:34:25.376 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.376 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:34:25.376 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:34:25.376 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:34:25.376 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.376 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:34:25.376 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.376 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:34:25.376 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:34:25.376 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:34:25.376 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.376 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.377 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.377 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:34:25.377 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.377 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.377 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:25.377 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:34:25.377 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:34:25.377 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:34:25.377 INFO:teuthology.orchestra.run.smithi190.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:34:25.377 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:34:25.377 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:34:25.377 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:34:25.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-test 2024-03-25T16:34:25.890 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:34:25.969 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:34:26.001 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:34:26.036 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:34:26.066 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:34:26.082 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:34:26.258 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:34:26.271 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:22 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:34:26.313 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:34:26.418 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:34:26.419 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:34:26.419 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-25T16:34:26.419 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:34:26.419 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:34:26.420 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:34:26.420 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-25T16:34:26.420 INFO:teuthology.orchestra.run.smithi190.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:34:26.420 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:34:26.420 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:34:26.420 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:34:26.420 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:34:26.420 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:34:26.420 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:34:26.420 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:34:26.420 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-03-25T16:34:26.420 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:34:26.421 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 48 M 2024-03-25T16:34:26.421 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 204 M 2024-03-25T16:34:26.421 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:34:26.631 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:34:26.696 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-25T16:34:26.737 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:26.746 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T16:34:26.796 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-25T16:34:26.838 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:26.927 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:34:26.963 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:34:27.026 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:34:27.055 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:34:27.080 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 661 kB/s | 187 kB 00:00 2024-03-25T16:34:27.111 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:34:27.184 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:34:27.208 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:34:27.254 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:34:27.281 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:34:27.347 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 203 kB/s | 163 kB 00:00 2024-03-25T16:34:27.389 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:34:27.432 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:34:27.473 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:34:27.505 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:34:27.526 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:34:27.576 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:34:27.886 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:34:27.922 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:34:28.011 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:34:28.043 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:34:28.084 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:34:28.538 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:34:28.566 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:34:28.638 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:34:28.714 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:34:28.736 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:34:30.854 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:30.907 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:31.455 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:31.494 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:34:31.585 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:34:31.585 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:34:31.585 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:34:31.585 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:34:31.620 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:40.961 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:40.962 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /sys 2024-03-25T16:34:40.962 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /proc 2024-03-25T16:34:40.962 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /mnt 2024-03-25T16:34:40.962 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /var/tmp 2024-03-25T16:34:40.962 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /home 2024-03-25T16:34:40.962 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /root 2024-03-25T16:34:40.962 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /tmp 2024-03-25T16:34:40.962 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:34:41.865 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:41.890 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:41.890 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:41.890 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:34:41.890 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:41.890 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:41.890 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:34:41.892 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:41.974 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:41.975 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:34:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:34:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:34:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:34:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:34:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:34:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:34:43.057 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:34:43.058 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:34:43.058 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:34:43.058 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:34:43.059 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:34:43.059 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:34:43.059 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:34:43.059 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:43.059 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:34:43.059 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:34:44.276 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:44.276 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:34:44.276 INFO:teuthology.orchestra.run.smithi099.stdout:Upgraded: 2024-03-25T16:34:44.276 INFO:teuthology.orchestra.run.smithi099.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.276 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.276 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:34:44.276 INFO:teuthology.orchestra.run.smithi099.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:34:44.276 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.276 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.276 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.276 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:34:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:34:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:34:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:34:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:34:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:34:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:34:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:34:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:34:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:34:44.277 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:34:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:34:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:34:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:34:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:34:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:34:44.278 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:34:44.278 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:34:44.579 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-test 2024-03-25T16:34:45.139 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:42 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:34:45.296 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:34:45.297 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:34:45.297 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-03-25T16:34:45.297 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:34:45.297 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:34:45.297 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:34:45.297 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-03-25T16:34:45.297 INFO:teuthology.orchestra.run.smithi099.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:34:45.297 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:34:45.298 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:34:45.298 INFO:teuthology.orchestra.run.smithi099.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:34:45.298 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:34:45.298 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:34:45.298 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:34:45.298 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:34:45.298 INFO:teuthology.orchestra.run.smithi099.stdout:Install 6 Packages 2024-03-25T16:34:45.298 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:34:45.299 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 48 M 2024-03-25T16:34:45.299 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 204 M 2024-03-25T16:34:45.299 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:34:45.502 INFO:teuthology.orchestra.run.smithi099.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-03-25T16:34:45.577 INFO:teuthology.orchestra.run.smithi099.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.6 MB/s | 300 kB 00:00 2024-03-25T16:34:45.611 INFO:teuthology.orchestra.run.smithi099.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T16:34:45.744 INFO:teuthology.orchestra.run.smithi099.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-25T16:34:46.420 INFO:teuthology.orchestra.run.smithi099.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 155 kB/s | 163 kB 00:01 2024-03-25T16:35:57.419 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 533 kB/s | 47 MB 01:30 2024-03-25T16:35:57.421 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:57.421 INFO:teuthology.orchestra.run.smithi190.stdout:Total 542 kB/s | 48 MB 01:31 2024-03-25T16:35:57.433 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:35:57.453 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:35:57.453 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:35:57.842 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:35:57.842 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:35:58.452 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:35:58.526 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:35:58.570 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:35:58.610 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:35:58.660 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:35:58.688 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:36:03.133 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:36:03.203 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:36:05.622 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:36:05.622 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:36:05.622 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:36:05.622 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:36:05.622 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:36:05.622 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:36:06.025 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:36:06.025 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:36:06.025 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:36:06.025 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:06.026 INFO:teuthology.orchestra.run.smithi190.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:36:06.026 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:06.026 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:36:06.026 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:36:06.026 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:36:06.026 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:36:06.026 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:36:06.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph 2024-03-25T16:36:06.887 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:03:02 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:36:07.046 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:36:07.049 INFO:teuthology.orchestra.run.smithi190.stdout:========================================================================================= 2024-03-25T16:36:07.049 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-25T16:36:07.049 INFO:teuthology.orchestra.run.smithi190.stdout:========================================================================================= 2024-03-25T16:36:07.049 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:36:07.049 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:36:07.049 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-25T16:36:07.049 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:36:07.049 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:36:07.049 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:36:07.050 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:36:07.050 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:36:07.050 INFO:teuthology.orchestra.run.smithi190.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:36:07.050 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:36:07.050 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:36:07.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:36:07.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:36:07.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:36:07.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:36:07.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:36:07.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:36:07.050 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:36:07.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:36:07.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:36:07.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:36:07.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:36:07.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:36:07.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:36:07.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:36:07.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:36:07.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:36:07.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:36:07.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:36:07.051 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:36:07.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:36:07.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:36:07.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:36:07.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:36:07.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:36:07.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:36:07.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:36:07.052 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:36:07.052 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:36:07.052 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:36:07.052 INFO:teuthology.orchestra.run.smithi190.stdout:========================================================================================= 2024-03-25T16:36:07.052 INFO:teuthology.orchestra.run.smithi190.stdout:Install 36 Packages 2024-03-25T16:36:07.053 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:36:07.053 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 30 M 2024-03-25T16:36:07.053 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 107 M 2024-03-25T16:36:07.053 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:36:08.017 INFO:teuthology.orchestra.run.smithi190.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-25T16:36:11.026 INFO:teuthology.orchestra.run.smithi190.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 448 kB/s | 1.5 MB 00:03 2024-03-25T16:36:13.351 INFO:teuthology.orchestra.run.smithi190.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 379 kB/s | 2.1 MB 00:05 2024-03-25T16:36:13.934 INFO:teuthology.orchestra.run.smithi190.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 424 kB/s | 247 kB 00:00 2024-03-25T16:36:14.059 INFO:teuthology.orchestra.run.smithi190.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T16:36:14.101 INFO:teuthology.orchestra.run.smithi190.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:36:14.135 INFO:teuthology.orchestra.run.smithi190.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 945 kB/s | 32 kB 00:00 2024-03-25T16:36:14.169 INFO:teuthology.orchestra.run.smithi190.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:36:14.219 INFO:teuthology.orchestra.run.smithi190.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-25T16:36:14.278 INFO:teuthology.orchestra.run.smithi190.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-25T16:36:14.503 INFO:teuthology.orchestra.run.smithi190.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 5.1 MB/s | 1.1 MB 00:00 2024-03-25T16:36:14.545 INFO:teuthology.orchestra.run.smithi190.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:36:14.595 INFO:teuthology.orchestra.run.smithi190.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T16:36:14.654 INFO:teuthology.orchestra.run.smithi190.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.2 MB/s | 187 kB 00:00 2024-03-25T16:36:14.862 INFO:teuthology.orchestra.run.smithi190.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 599 kB/s | 125 kB 00:00 2024-03-25T16:36:15.129 INFO:teuthology.orchestra.run.smithi190.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 84 kB/s | 22 kB 00:00 2024-03-25T16:36:15.396 INFO:teuthology.orchestra.run.smithi190.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 567 kB/s | 151 kB 00:00 2024-03-25T16:36:15.455 INFO:teuthology.orchestra.run.smithi190.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 746 kB/s | 43 kB 00:00 2024-03-25T16:36:15.538 INFO:teuthology.orchestra.run.smithi190.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-25T16:36:15.605 INFO:teuthology.orchestra.run.smithi190.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.2 MB/s | 358 kB 00:00 2024-03-25T16:36:15.697 INFO:teuthology.orchestra.run.smithi190.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 117 kB/s | 11 kB 00:00 2024-03-25T16:36:15.739 INFO:teuthology.orchestra.run.smithi190.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 424 kB/s | 18 kB 00:00 2024-03-25T16:36:15.781 INFO:teuthology.orchestra.run.smithi190.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 559 kB/s | 23 kB 00:00 2024-03-25T16:36:15.823 INFO:teuthology.orchestra.run.smithi190.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 466 kB/s | 19 kB 00:00 2024-03-25T16:36:15.865 INFO:teuthology.orchestra.run.smithi190.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 472 kB/s | 20 kB 00:00 2024-03-25T16:36:15.915 INFO:teuthology.orchestra.run.smithi190.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 926 kB/s | 46 kB 00:00 2024-03-25T16:36:15.957 INFO:teuthology.orchestra.run.smithi190.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-03-25T16:36:16.008 INFO:teuthology.orchestra.run.smithi190.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-25T16:36:16.058 INFO:teuthology.orchestra.run.smithi190.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-03-25T16:36:16.117 INFO:teuthology.orchestra.run.smithi190.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T16:36:16.175 INFO:teuthology.orchestra.run.smithi190.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 615 kB/s | 36 kB 00:00 2024-03-25T16:36:16.226 INFO:teuthology.orchestra.run.smithi190.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-25T16:36:16.276 INFO:teuthology.orchestra.run.smithi190.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.4 MB/s | 427 kB 00:00 2024-03-25T16:36:16.343 INFO:teuthology.orchestra.run.smithi190.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 300 kB/s | 20 kB 00:00 2024-03-25T16:36:19.510 INFO:teuthology.orchestra.run.smithi190.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 417 kB/s | 4.7 MB 00:11 2024-03-25T16:36:45.457 INFO:teuthology.orchestra.run.smithi190.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 499 kB/s | 17 MB 00:34 2024-03-25T16:36:45.462 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:45.462 INFO:teuthology.orchestra.run.smithi190.stdout:Total 791 kB/s | 30 MB 00:38 2024-03-25T16:36:45.486 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:36:45.526 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:36:45.526 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:36:45.954 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:36:45.954 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:36:47.280 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:36:47.334 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:36:47.380 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:36:47.411 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:36:47.481 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:36:47.519 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:36:47.551 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:36:47.635 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:36:47.705 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:36:47.745 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:36:47.817 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:36:47.871 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:36:47.910 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:36:47.959 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:36:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:36:48.197 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:36:48.248 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:36:48.286 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:36:48.321 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:36:48.351 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:36:48.426 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:36:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:36:48.503 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:36:48.544 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:36:48.579 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:36:48.621 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:36:48.711 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:36:48.753 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:36:48.825 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:36:48.961 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:36:49.078 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:36:49.244 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:36:49.274 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:36:49.300 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:36:49.300 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:49.300 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:36:49.300 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:36:49.300 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:36:49.300 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:36:50.759 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:36:50.787 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:36:50.788 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:50.788 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:36:50.788 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:36:50.788 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:36:50.788 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:36:51.227 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:36:51.250 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:36:51.250 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:51.250 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:36:51.250 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:36:51.250 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:36:51.250 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:36:51.460 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:36:51.483 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:36:51.484 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:51.484 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:36:51.484 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:36:51.484 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:36:51.484 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:36:51.549 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:36:53.231 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:36:53.231 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:36:53.231 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:36:53.232 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:36:53.232 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:36:53.232 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:36:53.232 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:36:53.232 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:36:53.232 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:36:53.232 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:36:53.232 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:36:53.232 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:36:53.232 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:36:53.232 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:36:53.232 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:36:53.232 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:36:53.232 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:36:53.233 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:36:53.233 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:36:53.233 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:36:53.233 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:36:53.233 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:36:53.233 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:36:53.233 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:36:53.233 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:36:53.233 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:36:53.233 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:36:53.233 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:36:53.233 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:36:53.234 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:36:53.234 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:36:53.234 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:36:53.234 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:36:53.235 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:36:53.235 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:36:53.235 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:36:54.214 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:36:54.214 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:36:54.214 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:36:54.215 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:36:54.216 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:36:54.217 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:36:54.217 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:36:54.217 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:36:54.217 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:36:54.217 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:36:54.217 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:36:54.217 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:36:54.217 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:36:54.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-base 2024-03-25T16:36:55.152 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:03:51 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:36:55.248 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:55.300 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:36:55.301 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-25T16:36:55.301 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:36:55.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install cephadm 2024-03-25T16:36:55.934 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:03:51 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:36:56.084 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:36:56.085 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:36:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-25T16:36:56.085 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:36:56.085 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:36:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:36:56.085 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:36:56.086 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:36:56.086 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:36:56.086 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T16:36:56.086 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:36:56.086 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 747 k 2024-03-25T16:36:56.086 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 754 k 2024-03-25T16:36:56.086 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:36:58.446 INFO:teuthology.orchestra.run.smithi190.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 317 kB/s | 747 kB 00:02 2024-03-25T16:36:58.447 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:58.447 INFO:teuthology.orchestra.run.smithi190.stdout:Total 316 kB/s | 747 kB 00:02 2024-03-25T16:36:58.448 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:36:58.454 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:36:58.454 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:36:58.460 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:36:58.460 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:36:58.540 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:36:59.034 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:36:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:36:59.492 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:36:59.752 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:36:59.752 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:36:59.752 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:36:59.752 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:59.752 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:36:59.752 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:36:59.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:37:00.542 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:03:56 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:37:00.694 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:37:00.694 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:37:00.695 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T16:37:00.695 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:37:00.695 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:37:00.695 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:37:00.695 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:00.695 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:37:00.695 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:37:00.695 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T16:37:00.695 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:00.696 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 145 k 2024-03-25T16:37:00.696 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 436 k 2024-03-25T16:37:00.696 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:37:01.603 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 160 kB/s | 145 kB 00:00 2024-03-25T16:37:01.603 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:01.603 INFO:teuthology.orchestra.run.smithi190.stdout:Total 160 kB/s | 145 kB 00:00 2024-03-25T16:37:01.604 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:37:01.611 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:37:01.611 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:37:01.666 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:37:01.666 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:37:01.800 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:37:01.853 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:37:01.877 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:37:01.877 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:01.878 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:37:01.878 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:02.867 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:37:02.867 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:02.867 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:37:02.867 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:02.867 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:02.867 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:37:03.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr 2024-03-25T16:37:03.699 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:03:59 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:37:03.799 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:37:03.854 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:37:03.854 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-25T16:37:03.855 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:37:03.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:37:04.493 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:04:00 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:37:04.643 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:37:04.644 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-03-25T16:37:04.644 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-25T16:37:04.644 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-03-25T16:37:04.644 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:37:04.644 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:37:04.644 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-25T16:37:04.644 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:37:04.645 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:37:04.645 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:37:04.645 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:37:04.645 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:37:04.645 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:37:04.645 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:37:04.645 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:37:04.645 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:37:04.645 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:37:04.645 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:37:04.645 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:04.645 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:37:04.645 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-03-25T16:37:04.646 INFO:teuthology.orchestra.run.smithi190.stdout:Install 12 Packages 2024-03-25T16:37:04.646 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:04.646 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 6.9 M 2024-03-25T16:37:04.646 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 26 M 2024-03-25T16:37:04.646 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:37:05.757 INFO:teuthology.orchestra.run.smithi190.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 32 kB/s | 15 kB 00:00 2024-03-25T16:37:05.907 INFO:teuthology.orchestra.run.smithi190.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-25T16:37:05.949 INFO:teuthology.orchestra.run.smithi190.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 5.6 MB/s | 238 kB 00:00 2024-03-25T16:37:06.016 INFO:teuthology.orchestra.run.smithi190.stdout:(4/12): ceph-grafana-dashboards-19.0.0-1439.g5d 43 kB/s | 31 kB 00:00 2024-03-25T16:37:06.208 INFO:teuthology.orchestra.run.smithi190.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.3 MB/s | 863 kB 00:00 2024-03-25T16:37:06.542 INFO:teuthology.orchestra.run.smithi190.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 58 kB/s | 19 kB 00:00 2024-03-25T16:37:06.725 INFO:teuthology.orchestra.run.smithi190.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 773 kB/s | 548 kB 00:00 2024-03-25T16:37:06.842 INFO:teuthology.orchestra.run.smithi190.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-03-25T16:37:06.942 INFO:teuthology.orchestra.run.smithi190.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 307 kB/s | 31 kB 00:00 2024-03-25T16:37:07.060 INFO:teuthology.orchestra.run.smithi190.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-25T16:37:07.143 INFO:teuthology.orchestra.run.smithi190.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.4 MB/s | 2.0 MB 00:00 2024-03-25T16:37:11.726 INFO:teuthology.orchestra.run.smithi190.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 283 kB/s | 1.8 MB 00:06 2024-03-25T16:37:11.731 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:11.731 INFO:teuthology.orchestra.run.smithi190.stdout:Total 991 kB/s | 6.9 MB 00:07 2024-03-25T16:37:11.751 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:37:11.771 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:37:11.771 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:37:11.955 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:37:11.956 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:37:12.467 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:37:12.543 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:37:12.578 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:37:12.624 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:37:12.664 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:37:12.767 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:37:12.898 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:37:13.108 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:37:13.161 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:37:13.189 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:37:13.220 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:37:13.461 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:37:13.495 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:37:14.118 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:37:14.118 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:37:14.118 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:37:14.119 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:37:14.119 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:37:14.119 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:37:14.119 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:37:14.119 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:37:14.119 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:37:14.119 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:37:14.119 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:37:14.119 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:37:14.464 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:37:14.464 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:14.465 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:37:14.465 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:37:14.465 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:14.465 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:14.465 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:14.465 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:37:14.465 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:37:14.465 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:37:14.465 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:37:14.465 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:37:14.465 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:37:14.465 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:37:14.465 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:37:14.465 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:14.466 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:37:14.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:37:15.302 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:37:15.453 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:37:15.454 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-03-25T16:37:15.454 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-25T16:37:15.454 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-03-25T16:37:15.454 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:37:15.454 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:37:15.454 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-25T16:37:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:37:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:37:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:37:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:37:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:37:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:37:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:37:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:37:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:37:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:37:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:37:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:15.455 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:37:15.455 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-03-25T16:37:15.456 INFO:teuthology.orchestra.run.smithi190.stdout:Install 12 Packages 2024-03-25T16:37:15.456 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:15.456 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 37 M 2024-03-25T16:37:15.456 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 209 M 2024-03-25T16:37:15.456 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:37:15.758 INFO:teuthology.orchestra.run.smithi190.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 258 kB/s | 32 kB 00:00 2024-03-25T16:37:15.841 INFO:teuthology.orchestra.run.smithi190.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 283 kB/s | 17 kB 00:00 2024-03-25T16:37:15.866 INFO:teuthology.orchestra.run.smithi190.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T16:37:16.034 INFO:teuthology.orchestra.run.smithi190.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-25T16:37:16.176 INFO:teuthology.orchestra.run.smithi190.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-25T16:37:16.226 INFO:teuthology.orchestra.run.smithi190.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.1 MB/s | 3.0 MB 00:00 2024-03-25T16:37:16.251 INFO:teuthology.orchestra.run.smithi190.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 217 kB 00:00 2024-03-25T16:37:16.602 INFO:teuthology.orchestra.run.smithi190.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-25T16:37:16.827 INFO:teuthology.orchestra.run.smithi190.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 869 kB/s | 195 kB 00:00 2024-03-25T16:37:16.977 INFO:teuthology.orchestra.run.smithi190.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-25T16:37:17.052 INFO:teuthology.orchestra.run.smithi190.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-25T16:37:18.147 INFO:teuthology.orchestra.run.smithi099.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 317 kB/s | 47 MB 02:32 2024-03-25T16:37:18.150 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:18.150 INFO:teuthology.orchestra.run.smithi099.stdout:Total 323 kB/s | 48 MB 02:32 2024-03-25T16:37:18.161 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:37:18.181 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:37:18.181 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:37:18.579 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:37:18.580 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:37:19.159 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:37:19.248 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:37:19.297 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:37:19.338 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:37:19.391 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:37:19.417 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:37:23.871 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:37:23.949 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:37:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:37:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:37:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:37:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:37:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:37:26.219 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:37:26.584 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:37:26.584 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:37:26.584 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:37:26.584 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:26.584 INFO:teuthology.orchestra.run.smithi099.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:37:26.584 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:26.585 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:37:26.585 INFO:teuthology.orchestra.run.smithi099.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:37:26.585 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:37:26.585 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:37:26.585 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:37:26.891 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph 2024-03-25T16:37:27.450 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:04:24 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:37:27.602 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:37:27.605 INFO:teuthology.orchestra.run.smithi099.stdout:========================================================================================= 2024-03-25T16:37:27.605 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-03-25T16:37:27.605 INFO:teuthology.orchestra.run.smithi099.stdout:========================================================================================= 2024-03-25T16:37:27.605 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:37:27.605 INFO:teuthology.orchestra.run.smithi099.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:37:27.605 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-03-25T16:37:27.605 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:37:27.605 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:37:27.605 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:37:27.605 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:37:27.606 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:37:27.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:37:27.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:37:27.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:37:27.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:37:27.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:37:27.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:37:27.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:37:27.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:37:27.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:37:27.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:37:27.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:37:27.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:37:27.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:37:27.607 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:37:27.608 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:37:27.608 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:37:27.608 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:37:27.608 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:37:27.608 INFO:teuthology.orchestra.run.smithi099.stdout:========================================================================================= 2024-03-25T16:37:27.608 INFO:teuthology.orchestra.run.smithi099.stdout:Install 36 Packages 2024-03-25T16:37:27.608 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:37:27.608 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 30 M 2024-03-25T16:37:27.609 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 107 M 2024-03-25T16:37:27.609 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:37:29.066 INFO:teuthology.orchestra.run.smithi099.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 10 kB/s | 6.6 kB 00:00 2024-03-25T16:37:34.223 INFO:teuthology.orchestra.run.smithi099.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 262 kB/s | 1.5 MB 00:05 2024-03-25T16:37:38.181 INFO:teuthology.orchestra.run.smithi099.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 222 kB/s | 2.1 MB 00:09 2024-03-25T16:37:39.515 INFO:teuthology.orchestra.run.smithi099.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 186 kB/s | 247 kB 00:01 2024-03-25T16:37:39.698 INFO:teuthology.orchestra.run.smithi099.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-25T16:37:39.732 INFO:teuthology.orchestra.run.smithi099.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-25T16:37:39.765 INFO:teuthology.orchestra.run.smithi099.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 951 kB/s | 32 kB 00:00 2024-03-25T16:37:39.799 INFO:teuthology.orchestra.run.smithi099.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:37:39.832 INFO:teuthology.orchestra.run.smithi099.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:37:39.874 INFO:teuthology.orchestra.run.smithi099.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:37:39.958 INFO:teuthology.orchestra.run.smithi099.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-25T16:37:39.991 INFO:teuthology.orchestra.run.smithi099.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:37:40.025 INFO:teuthology.orchestra.run.smithi099.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-25T16:37:40.058 INFO:teuthology.orchestra.run.smithi099.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T16:37:40.192 INFO:teuthology.orchestra.run.smithi099.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 937 kB/s | 125 kB 00:00 2024-03-25T16:37:40.759 INFO:teuthology.orchestra.run.smithi099.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-03-25T16:37:41.018 INFO:teuthology.orchestra.run.smithi099.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 586 kB/s | 151 kB 00:00 2024-03-25T16:37:41.077 INFO:teuthology.orchestra.run.smithi099.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 744 kB/s | 43 kB 00:00 2024-03-25T16:37:41.177 INFO:teuthology.orchestra.run.smithi099.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-25T16:37:41.277 INFO:teuthology.orchestra.run.smithi099.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-03-25T16:37:41.336 INFO:teuthology.orchestra.run.smithi099.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-25T16:37:41.402 INFO:teuthology.orchestra.run.smithi099.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 266 kB/s | 18 kB 00:00 2024-03-25T16:37:41.487 INFO:teuthology.orchestra.run.smithi099.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 275 kB/s | 23 kB 00:00 2024-03-25T16:37:41.545 INFO:teuthology.orchestra.run.smithi099.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 337 kB/s | 19 kB 00:00 2024-03-25T16:37:41.603 INFO:teuthology.orchestra.run.smithi099.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 338 kB/s | 20 kB 00:00 2024-03-25T16:37:41.662 INFO:teuthology.orchestra.run.smithi099.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 795 kB/s | 46 kB 00:00 2024-03-25T16:37:41.721 INFO:teuthology.orchestra.run.smithi099.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T16:37:41.787 INFO:teuthology.orchestra.run.smithi099.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-25T16:37:41.846 INFO:teuthology.orchestra.run.smithi099.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2024-03-25T16:37:41.905 INFO:teuthology.orchestra.run.smithi099.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T16:37:41.972 INFO:teuthology.orchestra.run.smithi099.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 537 kB/s | 36 kB 00:00 2024-03-25T16:37:42.030 INFO:teuthology.orchestra.run.smithi099.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.9 MB/s | 230 kB 00:00 2024-03-25T16:37:42.122 INFO:teuthology.orchestra.run.smithi099.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.6 MB/s | 427 kB 00:00 2024-03-25T16:37:42.181 INFO:teuthology.orchestra.run.smithi099.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-25T16:37:47.215 INFO:teuthology.orchestra.run.smithi190.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 240 kB/s | 7.4 MB 00:31 2024-03-25T16:37:47.219 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:47.219 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.2 MB/s | 37 MB 00:31 2024-03-25T16:37:47.264 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:37:47.289 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:37:47.290 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:37:47.472 INFO:teuthology.orchestra.run.smithi099.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 261 kB/s | 4.7 MB 00:18 2024-03-25T16:37:47.648 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:37:47.648 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:37:48.356 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:37:48.421 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:37:48.452 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:37:48.546 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:37:48.606 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:37:48.784 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:37:48.814 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:37:48.897 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:37:49.549 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:37:49.668 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:37:50.933 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:37:51.976 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:37:52.005 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:37:54.565 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:37:54.565 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:37:54.565 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:37:54.565 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:37:54.565 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:37:54.565 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:37:54.565 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:37:54.565 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:37:54.565 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:37:54.566 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:37:54.566 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:37:54.566 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:37:54.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:37:54.990 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:54.990 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:37:54.990 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:37:54.990 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:37:54.990 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:37:54.990 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:37:54.991 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:37:54.991 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:37:54.991 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:37:54.991 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:37:54.991 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:37:54.991 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:37:54.991 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:37:54.991 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:37:54.991 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:54.991 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:37:55.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:37:55.974 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:04:51 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:37:56.124 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:37:56.125 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-25T16:37:56.126 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-25T16:37:56.126 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-25T16:37:56.126 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:37:56.126 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:37:56.126 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-25T16:37:56.126 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:37:56.126 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:37:56.126 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:37:56.126 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:37:56.126 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:37:56.126 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:37:56.126 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:37:56.127 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:37:56.127 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:37:56.127 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:37:56.127 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:37:56.127 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:37:56.127 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:56.127 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:37:56.127 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-25T16:37:56.127 INFO:teuthology.orchestra.run.smithi190.stdout:Install 13 Packages 2024-03-25T16:37:56.127 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:37:56.128 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 2.0 M 2024-03-25T16:37:56.128 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 26 M 2024-03-25T16:37:56.128 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:37:56.281 INFO:teuthology.orchestra.run.smithi190.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 224 kB/s | 24 kB 00:00 2024-03-25T16:37:56.323 INFO:teuthology.orchestra.run.smithi190.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 120 kB/s | 18 kB 00:00 2024-03-25T16:37:56.373 INFO:teuthology.orchestra.run.smithi190.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-25T16:37:56.399 INFO:teuthology.orchestra.run.smithi190.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:37:56.432 INFO:teuthology.orchestra.run.smithi190.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T16:37:56.449 INFO:teuthology.orchestra.run.smithi190.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 866 kB/s | 43 kB 00:00 2024-03-25T16:37:56.533 INFO:teuthology.orchestra.run.smithi190.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 179 kB/s | 15 kB 00:00 2024-03-25T16:37:56.550 INFO:teuthology.orchestra.run.smithi190.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 274 kB/s | 32 kB 00:00 2024-03-25T16:37:56.675 INFO:teuthology.orchestra.run.smithi190.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.4 MB/s | 203 kB 00:00 2024-03-25T16:37:56.725 INFO:teuthology.orchestra.run.smithi190.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-25T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-03-25T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout:(12/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 83 kB/s | 52 kB 00:00 2024-03-25T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.3 MB/s | 1.0 MB 00:00 2024-03-25T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-03-25T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:37:56.875 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:37:56.875 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:37:56.964 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:37:56.972 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:37:57.488 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:37:57.664 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:37:57.720 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:37:57.782 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:37:57.828 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:37:57.870 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:37:57.963 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:37:58.064 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:37:58.120 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:37:58.835 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:37:58.877 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:37:58.911 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:37:58.967 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:37:58.989 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:37:59.773 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:37:59.773 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:37:59.773 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:37:59.773 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:37:59.773 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:37:59.774 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:37:59.774 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:37:59.774 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:37:59.774 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:37:59.774 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:37:59.774 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:37:59.774 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:37:59.774 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:38:00.127 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:38:00.127 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:00.127 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:38:00.127 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:00.127 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:38:00.127 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:38:00.127 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:38:00.127 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:38:00.128 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:38:00.128 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:38:00.128 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:38:00.128 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:38:00.128 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:38:00.128 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:38:00.128 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:38:00.128 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:38:00.128 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:00.128 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:00.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:38:00.956 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:04:56 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:01.107 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:01.108 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-25T16:38:01.108 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-25T16:38:01.108 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-25T16:38:01.108 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:38:01.108 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:38:01.108 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-25T16:38:01.108 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:38:01.108 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:38:01.108 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:38:01.108 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:38:01.108 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:38:01.108 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:01.109 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:38:01.109 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-25T16:38:01.109 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-03-25T16:38:01.109 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:01.109 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 6.8 M 2024-03-25T16:38:01.109 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 33 M 2024-03-25T16:38:01.110 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:38:02.312 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 261 kB/s | 248 kB 00:00 2024-03-25T16:38:02.496 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.1 MB/s | 5.8 MB 00:01 2024-03-25T16:38:02.763 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 217 kB/s | 58 kB 00:00 2024-03-25T16:38:02.796 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-25T16:38:02.855 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 556 kB/s | 51 kB 00:00 2024-03-25T16:38:02.913 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 89 kB/s | 138 kB 00:01 2024-03-25T16:38:02.917 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:02.917 INFO:teuthology.orchestra.run.smithi190.stdout:Total 3.8 MB/s | 6.8 MB 00:01 2024-03-25T16:38:02.930 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:38:02.940 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:38:02.941 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:38:03.029 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:38:03.030 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:38:03.321 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:38:03.481 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:38:03.532 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:38:04.056 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:38:04.135 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:38:04.196 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:38:04.219 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:38:05.106 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:38:05.106 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:38:05.106 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:38:05.106 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:38:05.106 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:38:05.106 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:38:05.417 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:38:05.417 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:05.417 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:38:05.417 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:05.417 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:38:05.417 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:38:05.417 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:38:05.417 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:38:05.417 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:38:05.417 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:05.418 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:05.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-fuse 2024-03-25T16:38:06.249 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:02 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:06.401 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:06.401 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:06.402 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-25T16:38:06.402 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:06.402 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:38:06.402 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:38:06.402 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-25T16:38:06.402 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:38:06.402 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:06.402 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:38:06.402 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:06.402 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-03-25T16:38:06.402 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:06.403 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 964 k 2024-03-25T16:38:06.403 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 2.8 M 2024-03-25T16:38:06.403 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:38:06.515 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 703 kB/s | 79 kB 00:00 2024-03-25T16:38:10.723 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 205 kB/s | 885 kB 00:04 2024-03-25T16:38:10.724 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:10.725 INFO:teuthology.orchestra.run.smithi190.stdout:Total 223 kB/s | 964 kB 00:04 2024-03-25T16:38:10.725 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:38:10.748 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:38:10.748 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:38:10.824 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:38:10.824 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:38:10.999 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:38:11.144 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:38:11.208 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:38:11.918 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:38:11.918 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:38:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:38:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:12.257 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:38:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:38:12.257 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:12.257 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:12.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-volume 2024-03-25T16:38:13.087 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:13.237 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:13.238 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:13.238 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-25T16:38:13.238 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:13.238 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:38:13.238 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:38:13.238 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-25T16:38:13.238 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:38:13.238 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:13.238 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:38:13.238 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:13.239 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-03-25T16:38:13.239 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:13.239 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 332 k 2024-03-25T16:38:13.239 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 1.5 M 2024-03-25T16:38:13.239 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:38:13.355 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 598 kB/s | 69 kB 00:00 2024-03-25T16:38:14.330 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 242 kB/s | 264 kB 00:01 2024-03-25T16:38:14.331 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:14.332 INFO:teuthology.orchestra.run.smithi190.stdout:Total 304 kB/s | 332 kB 00:01 2024-03-25T16:38:14.332 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:38:14.342 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:38:14.342 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:38:14.386 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:38:14.386 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:38:14.538 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:38:14.712 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:38:14.766 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:38:14.782 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:38:14.782 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:14.782 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:38:14.782 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:15.527 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:38:15.902 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:38:15.902 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:15.902 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:38:15.902 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:15.902 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:38:15.903 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:15.903 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:16.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados-devel 2024-03-25T16:38:16.815 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:16.964 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:16.965 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:16.965 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T16:38:16.965 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:16.965 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:38:16.965 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:38:16.965 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:16.965 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:38:16.965 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:16.965 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T16:38:16.965 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:16.966 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 127 k 2024-03-25T16:38:16.966 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 456 k 2024-03-25T16:38:16.966 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:38:18.236 INFO:teuthology.orchestra.run.smithi190.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 100 kB/s | 127 kB 00:01 2024-03-25T16:38:18.237 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:18.237 INFO:teuthology.orchestra.run.smithi190.stdout:Total 100 kB/s | 127 kB 00:01 2024-03-25T16:38:18.238 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:38:18.244 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:38:18.244 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:38:18.304 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:38:18.304 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:38:18.449 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:38:18.562 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:19.015 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:19.313 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:38:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:19.313 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:19.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs2 2024-03-25T16:38:20.130 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:20.230 INFO:teuthology.orchestra.run.smithi190.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:20.286 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:20.286 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-25T16:38:20.287 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:20.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs-devel 2024-03-25T16:38:20.926 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:21.078 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:21.079 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:21.079 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T16:38:21.079 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:21.079 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:38:21.080 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:38:21.080 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:21.080 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:38:21.080 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:21.080 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T16:38:21.080 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:21.080 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 31 k 2024-03-25T16:38:21.080 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 140 k 2024-03-25T16:38:21.080 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 60 kB/s | 31 kB 00:00 2024-03-25T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout:Total 60 kB/s | 31 kB 00:00 2024-03-25T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:38:21.753 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:38:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:22.255 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:22.539 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:22.539 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:22.539 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:38:22.539 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:22.539 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:22.539 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:22.699 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados2 2024-03-25T16:38:23.275 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:23.380 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:23.435 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:23.436 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-25T16:38:23.436 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:23.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librbd1 2024-03-25T16:38:24.109 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:20 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:24.221 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:24.277 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:24.278 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-25T16:38:24.278 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:24.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rados 2024-03-25T16:38:24.900 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:20 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:24.996 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:25.048 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:25.049 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-25T16:38:25.049 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:25.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rgw 2024-03-25T16:38:25.669 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:21 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:25.765 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:25.816 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:25.817 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-25T16:38:25.817 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:25.859 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-cephfs 2024-03-25T16:38:26.427 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:22 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:26.523 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:26.575 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:26.576 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-25T16:38:26.576 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:26.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rbd 2024-03-25T16:38:27.186 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:23 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:27.282 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:27.333 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:27.334 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-25T16:38:27.334 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:27.387 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-fuse 2024-03-25T16:38:27.952 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:23 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:28.100 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:28.101 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:28.101 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-25T16:38:28.101 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:28.101 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:38:28.101 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:38:28.101 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:28.101 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:38:28.101 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:28.101 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T16:38:28.102 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:28.102 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 85 k 2024-03-25T16:38:28.102 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 231 k 2024-03-25T16:38:28.102 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:38:29.110 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 85 kB/s | 85 kB 00:01 2024-03-25T16:38:29.111 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:29.111 INFO:teuthology.orchestra.run.smithi190.stdout:Total 84 kB/s | 85 kB 00:01 2024-03-25T16:38:29.111 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:38:29.119 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:38:29.120 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:38:29.177 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:38:29.178 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:38:29.317 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:38:29.409 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:29.809 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:30.113 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:30.113 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:30.113 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:38:30.113 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:30.113 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:30.113 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:30.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-mirror 2024-03-25T16:38:30.894 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:26 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:31.048 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:31.049 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:31.049 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T16:38:31.049 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:31.049 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:38:31.049 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:38:31.049 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:31.049 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:38:31.049 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:31.049 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T16:38:31.049 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:31.050 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.1 M 2024-03-25T16:38:31.050 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 13 M 2024-03-25T16:38:31.050 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:38:40.055 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 351 kB/s | 3.1 MB 00:09 2024-03-25T16:38:40.056 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:40.056 INFO:teuthology.orchestra.run.smithi190.stdout:Total 351 kB/s | 3.1 MB 00:09 2024-03-25T16:38:40.056 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:38:40.067 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:38:40.067 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:38:40.144 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:38:40.144 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:38:40.572 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:38:40.638 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:40.664 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:40.664 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:40.664 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:38:40.664 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:38:40.664 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:38:40.664 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:40.953 INFO:teuthology.orchestra.run.smithi099.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 257 kB/s | 17 MB 01:06 2024-03-25T16:38:40.957 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:40.958 INFO:teuthology.orchestra.run.smithi099.stdout:Total 414 kB/s | 30 MB 01:13 2024-03-25T16:38:40.978 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:38:41.021 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:38:41.021 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:38:41.442 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:38:41.442 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:38:41.879 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:41.879 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:41.879 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:38:41.879 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:41.879 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:41.879 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:42.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-nbd 2024-03-25T16:38:42.699 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:42.808 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:38:42.848 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:42.849 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:42.849 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-25T16:38:42.849 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:42.849 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:38:42.849 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:38:42.849 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:42.849 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:38:42.849 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:42.849 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T16:38:42.850 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:42.850 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 171 k 2024-03-25T16:38:42.850 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 494 k 2024-03-25T16:38:42.850 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:38:42.871 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:38:42.909 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:38:42.941 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:38:43.010 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:38:43.056 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:38:43.089 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:38:43.179 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:38:43.266 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:38:43.310 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:38:43.379 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:38:43.434 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:38:43.472 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:38:43.529 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:38:43.583 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:38:43.769 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:38:43.851 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:38:43.890 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:38:43.925 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:38:43.954 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:38:44.008 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 148 kB/s | 171 kB 00:01 2024-03-25T16:38:44.009 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:44.009 INFO:teuthology.orchestra.run.smithi190.stdout:Total 148 kB/s | 171 kB 00:01 2024-03-25T16:38:44.009 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:38:44.018 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:38:44.018 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:38:44.030 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:38:44.064 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:38:44.079 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:38:44.079 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:38:44.098 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:38:44.138 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:38:44.173 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:38:44.208 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:38:44.234 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:38:44.296 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:38:44.338 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:38:44.340 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:44.410 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:38:44.554 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:38:44.680 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:38:44.784 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:44.864 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:38:44.893 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:38:44.917 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:38:44.917 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:44.917 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:38:44.917 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:38:44.917 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:38:44.918 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:45.110 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:38:45.110 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:45.110 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:38:45.110 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:45.110 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:45.110 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:38:45.335 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:38:45.906 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:05:41 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:38:46.059 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:38:46.059 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:46.059 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T16:38:46.059 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:46.059 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:38:46.060 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:38:46.060 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-25T16:38:46.060 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:38:46.060 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:46.060 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:38:46.060 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:38:46.060 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-03-25T16:38:46.060 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:38:46.061 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 91 M 2024-03-25T16:38:46.061 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 498 M 2024-03-25T16:38:46.061 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:38:46.361 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:38:46.386 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:38:46.386 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:46.386 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:38:46.386 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:38:46.386 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:38:46.386 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:46.819 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:38:46.842 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:38:46.842 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:46.842 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:38:46.842 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:38:46.842 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:38:46.842 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:47.045 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:38:47.067 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:38:47.067 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:47.067 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:38:47.068 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:38:47.068 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:38:47.068 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:47.126 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:38:48.816 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:38:48.816 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:38:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:38:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:38:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:38:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:38:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:38:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:38:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:38:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:38:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:38:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:38:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:38:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:38:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:38:48.817 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:38:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:38:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:38:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:38:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:38:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:38:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:38:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:38:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:38:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:38:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:38:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:38:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:38:48.818 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:38:48.819 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:38:48.819 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:38:48.819 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:38:48.819 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:38:48.819 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:38:48.819 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:38:48.819 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:38:49.762 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:38:49.762 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:49.762 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:38:49.762 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:49.762 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:49.762 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:49.762 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:38:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:38:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:38:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:38:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:38:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:38:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:38:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:38:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:38:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:38:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:38:49.763 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:38:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:38:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:38:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:38:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:38:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:38:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:38:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:38:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:38:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:38:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:38:49.764 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:38:49.765 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:38:49.765 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:38:49.765 INFO:teuthology.orchestra.run.smithi099.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:38:49.765 INFO:teuthology.orchestra.run.smithi099.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:38:49.765 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:38:49.765 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:38:49.765 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:38:49.765 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:49.765 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:38:50.185 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-base 2024-03-25T16:38:50.763 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:05:47 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:38:50.860 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:50.911 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:38:50.912 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-03-25T16:38:50.912 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:38:50.974 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install cephadm 2024-03-25T16:38:51.531 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:05:48 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:38:51.680 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:38:51.681 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:38:51.681 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-03-25T16:38:51.681 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:38:51.681 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:38:51.681 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:38:51.681 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:51.682 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:38:51.682 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:38:51.682 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T16:38:51.682 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:51.682 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 747 k 2024-03-25T16:38:51.682 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 754 k 2024-03-25T16:38:51.682 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:38:54.075 INFO:teuthology.orchestra.run.smithi099.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 312 kB/s | 747 kB 00:02 2024-03-25T16:38:54.075 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:54.076 INFO:teuthology.orchestra.run.smithi099.stdout:Total 312 kB/s | 747 kB 00:02 2024-03-25T16:38:54.076 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:38:54.083 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:38:54.083 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:38:54.089 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:38:54.090 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:38:54.161 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:38:54.654 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:54.731 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:55.119 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:55.399 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:55.399 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:55.399 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:38:55.400 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:55.400 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:55.400 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:38:55.617 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:38:56.194 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:05:53 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:38:56.346 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:38:56.347 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:38:56.347 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T16:38:56.347 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:38:56.347 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:38:56.347 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:38:56.347 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:56.347 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:38:56.347 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:38:56.347 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T16:38:56.347 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:56.348 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 145 k 2024-03-25T16:38:56.348 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 436 k 2024-03-25T16:38:56.348 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:38:57.747 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 104 kB/s | 145 kB 00:01 2024-03-25T16:38:57.748 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:57.748 INFO:teuthology.orchestra.run.smithi099.stdout:Total 104 kB/s | 145 kB 00:01 2024-03-25T16:38:57.748 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:38:57.757 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:38:57.757 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:38:57.813 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:38:57.813 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:38:57.967 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:38:58.048 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:38:58.072 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:38:58.073 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:58.073 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:38:58.073 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:59.072 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:38:59.072 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:59.072 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:38:59.072 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:59.072 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:38:59.072 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:38:59.317 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr 2024-03-25T16:38:59.890 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:05:56 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:38:59.986 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:00.038 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:39:00.039 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-03-25T16:39:00.039 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:39:00.103 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:39:00.663 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:05:57 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:39:00.811 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:39:00.812 INFO:teuthology.orchestra.run.smithi099.stdout:====================================================================================== 2024-03-25T16:39:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-03-25T16:39:00.812 INFO:teuthology.orchestra.run.smithi099.stdout:====================================================================================== 2024-03-25T16:39:00.812 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:39:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:39:00.812 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-03-25T16:39:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:39:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:39:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:39:00.812 INFO:teuthology.orchestra.run.smithi099.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:39:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:39:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:39:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:39:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:39:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:39:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:39:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:39:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:00.813 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:39:00.813 INFO:teuthology.orchestra.run.smithi099.stdout:====================================================================================== 2024-03-25T16:39:00.813 INFO:teuthology.orchestra.run.smithi099.stdout:Install 12 Packages 2024-03-25T16:39:00.813 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:00.814 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 6.9 M 2024-03-25T16:39:00.814 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 26 M 2024-03-25T16:39:00.814 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:39:01.592 INFO:teuthology.orchestra.run.smithi099.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 33 kB/s | 15 kB 00:00 2024-03-25T16:39:01.700 INFO:teuthology.orchestra.run.smithi099.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 55 kB/s | 31 kB 00:00 2024-03-25T16:39:01.842 INFO:teuthology.orchestra.run.smithi099.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-03-25T16:39:01.867 INFO:teuthology.orchestra.run.smithi099.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 238 kB 00:00 2024-03-25T16:39:02.034 INFO:teuthology.orchestra.run.smithi099.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-03-25T16:39:02.468 INFO:teuthology.orchestra.run.smithi099.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 913 kB/s | 548 kB 00:00 2024-03-25T16:39:02.493 INFO:teuthology.orchestra.run.smithi099.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 42 kB/s | 19 kB 00:00 2024-03-25T16:39:03.227 INFO:teuthology.orchestra.run.smithi099.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 2.7 MB/s | 2.0 MB 00:00 2024-03-25T16:39:03.269 INFO:teuthology.orchestra.run.smithi099.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 186 kB/s | 144 kB 00:00 2024-03-25T16:39:03.302 INFO:teuthology.orchestra.run.smithi099.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 411 kB/s | 31 kB 00:00 2024-03-25T16:39:03.361 INFO:teuthology.orchestra.run.smithi099.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-25T16:39:09.752 INFO:teuthology.orchestra.run.smithi099.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 212 kB/s | 1.8 MB 00:08 2024-03-25T16:39:09.757 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:09.757 INFO:teuthology.orchestra.run.smithi099.stdout:Total 785 kB/s | 6.9 MB 00:08 2024-03-25T16:39:09.776 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:39:09.795 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:39:09.795 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:39:09.967 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:39:09.967 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:39:10.481 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:39:10.560 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:39:10.595 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:39:10.641 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:39:10.673 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:39:10.780 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:39:10.905 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:39:10.991 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:39:11.035 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:39:11.072 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:39:11.104 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:39:11.347 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:39:11.396 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:39:12.155 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:39:12.155 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:39:12.155 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:39:12.155 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:39:12.155 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:39:12.155 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:39:12.155 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:39:12.155 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:39:12.155 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:39:12.156 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:39:12.156 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:39:12.156 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:39:12.529 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:39:12.529 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:12.529 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:39:12.529 INFO:teuthology.orchestra.run.smithi099.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:39:12.529 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:12.529 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:12.529 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:12.529 INFO:teuthology.orchestra.run.smithi099.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:39:12.530 INFO:teuthology.orchestra.run.smithi099.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:39:12.530 INFO:teuthology.orchestra.run.smithi099.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:39:12.530 INFO:teuthology.orchestra.run.smithi099.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:39:12.530 INFO:teuthology.orchestra.run.smithi099.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:39:12.530 INFO:teuthology.orchestra.run.smithi099.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:39:12.530 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:39:12.530 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:39:12.530 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:12.530 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:39:12.787 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:39:13.346 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:06:10 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:39:13.496 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:39:13.497 INFO:teuthology.orchestra.run.smithi099.stdout:============================================================================================ 2024-03-25T16:39:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-03-25T16:39:13.497 INFO:teuthology.orchestra.run.smithi099.stdout:============================================================================================ 2024-03-25T16:39:13.497 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:39:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:39:13.497 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-03-25T16:39:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:39:13.497 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:39:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:39:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:39:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:39:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:39:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:39:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:39:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:39:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:39:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:39:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:13.498 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:39:13.498 INFO:teuthology.orchestra.run.smithi099.stdout:============================================================================================ 2024-03-25T16:39:13.498 INFO:teuthology.orchestra.run.smithi099.stdout:Install 12 Packages 2024-03-25T16:39:13.498 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:13.499 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 37 M 2024-03-25T16:39:13.499 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 209 M 2024-03-25T16:39:13.499 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:39:14.057 INFO:teuthology.orchestra.run.smithi099.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 223 kB/s | 32 kB 00:00 2024-03-25T16:39:14.082 INFO:teuthology.orchestra.run.smithi099.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 701 kB/s | 17 kB 00:00 2024-03-25T16:39:14.116 INFO:teuthology.orchestra.run.smithi099.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T16:39:14.166 INFO:teuthology.orchestra.run.smithi099.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-25T16:39:14.383 INFO:teuthology.orchestra.run.smithi099.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-25T16:39:14.450 INFO:teuthology.orchestra.run.smithi099.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-25T16:39:14.475 INFO:teuthology.orchestra.run.smithi099.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T16:39:14.851 INFO:teuthology.orchestra.run.smithi099.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-25T16:39:15.309 INFO:teuthology.orchestra.run.smithi099.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 966 kB/s | 803 kB 00:00 2024-03-25T16:39:15.401 INFO:teuthology.orchestra.run.smithi099.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 355 kB/s | 195 kB 00:00 2024-03-25T16:39:15.493 INFO:teuthology.orchestra.run.smithi099.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-25T16:39:28.150 INFO:teuthology.orchestra.run.smithi099.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 532 kB/s | 7.4 MB 00:14 2024-03-25T16:39:28.152 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:28.152 INFO:teuthology.orchestra.run.smithi099.stdout:Total 2.5 MB/s | 37 MB 00:14 2024-03-25T16:39:28.197 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:39:28.221 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:39:28.221 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:39:28.581 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:39:28.581 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:39:29.279 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:39:29.356 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:39:29.387 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:39:29.479 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:39:29.532 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:39:29.710 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:39:29.739 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:39:29.825 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:39:30.480 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:39:30.601 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:39:31.845 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:39:32.871 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:39:32.904 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:39:35.528 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:39:35.528 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:39:35.528 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:39:35.528 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:39:35.528 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:39:35.528 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:39:35.528 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:39:35.528 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:39:35.528 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:39:35.528 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:39:35.528 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:39:35.528 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:39:35.960 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:39:35.961 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:35.961 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:39:35.961 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:35.961 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:39:35.961 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:39:35.961 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:39:35.961 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:39:35.961 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:39:35.961 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:39:35.961 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:39:35.961 INFO:teuthology.orchestra.run.smithi099.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:39:35.961 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:39:35.961 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:39:35.962 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:39:35.962 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:35.962 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:39:36.366 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:39:36.940 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:06:33 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:39:37.089 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:39:37.091 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2024-03-25T16:39:37.091 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-03-25T16:39:37.091 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2024-03-25T16:39:37.091 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:39:37.091 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:39:37.091 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-03-25T16:39:37.091 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:39:37.091 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:39:37.091 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:39:37.091 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:39:37.091 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:39:37.092 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:39:37.092 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:39:37.092 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:39:37.092 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:39:37.092 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:39:37.092 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:39:37.092 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:39:37.092 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:37.092 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:39:37.092 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2024-03-25T16:39:37.092 INFO:teuthology.orchestra.run.smithi099.stdout:Install 13 Packages 2024-03-25T16:39:37.092 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:37.093 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 2.0 M 2024-03-25T16:39:37.093 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 26 M 2024-03-25T16:39:37.093 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:39:37.246 INFO:teuthology.orchestra.run.smithi099.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 227 kB/s | 24 kB 00:00 2024-03-25T16:39:37.288 INFO:teuthology.orchestra.run.smithi099.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 121 kB/s | 18 kB 00:00 2024-03-25T16:39:37.330 INFO:teuthology.orchestra.run.smithi099.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.8 MB/s | 174 kB 00:00 2024-03-25T16:39:37.347 INFO:teuthology.orchestra.run.smithi099.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-25T16:39:37.372 INFO:teuthology.orchestra.run.smithi099.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-03-25T16:39:37.389 INFO:teuthology.orchestra.run.smithi099.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T16:39:37.606 INFO:teuthology.orchestra.run.smithi099.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 138 kB/s | 32 kB 00:00 2024-03-25T16:39:37.623 INFO:teuthology.orchestra.run.smithi099.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-03-25T16:39:37.740 INFO:teuthology.orchestra.run.smithi099.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 87 kB/s | 52 kB 00:00 2024-03-25T16:39:37.773 INFO:teuthology.orchestra.run.smithi099.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.2 MB/s | 203 kB 00:00 2024-03-25T16:39:37.849 INFO:teuthology.orchestra.run.smithi099.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-03-25T16:39:37.924 INFO:teuthology.orchestra.run.smithi099.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.4 MB/s | 1.0 MB 00:00 2024-03-25T16:39:38.007 INFO:teuthology.orchestra.run.smithi099.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 221 kB/s | 59 kB 00:00 2024-03-25T16:39:38.011 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:38.011 INFO:teuthology.orchestra.run.smithi099.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-25T16:39:38.012 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:39:38.024 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:39:38.025 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:39:38.114 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:39:38.115 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:39:38.626 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:39:38.804 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:39:38.858 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:39:38.913 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:39:38.942 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:39:38.983 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:39:39.077 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:39:39.202 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:39:39.251 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:39:39.965 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:39:40.015 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:39:40.049 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:39:40.097 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:39:40.119 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:39:40.873 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:39:40.873 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:39:40.873 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:39:40.873 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:39:40.873 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:39:40.873 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:39:40.873 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:39:40.874 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:39:40.874 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:39:40.874 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:39:40.874 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:39:40.874 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:39:40.874 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:39:41.249 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:39:41.249 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:41.249 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:39:41.249 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:39:41.502 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:39:42.063 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:06:39 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:39:42.213 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:39:42.214 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2024-03-25T16:39:42.214 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-03-25T16:39:42.214 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2024-03-25T16:39:42.214 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:39:42.215 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:39:42.215 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-03-25T16:39:42.215 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:39:42.215 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:39:42.215 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:39:42.215 INFO:teuthology.orchestra.run.smithi099.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:39:42.215 INFO:teuthology.orchestra.run.smithi099.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:39:42.215 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:42.215 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:39:42.215 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2024-03-25T16:39:42.215 INFO:teuthology.orchestra.run.smithi099.stdout:Install 6 Packages 2024-03-25T16:39:42.215 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:42.216 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 6.8 M 2024-03-25T16:39:42.216 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 33 M 2024-03-25T16:39:42.216 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:39:42.808 INFO:teuthology.orchestra.run.smithi099.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T16:39:42.824 INFO:teuthology.orchestra.run.smithi099.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 839 kB/s | 248 kB 00:00 2024-03-25T16:39:42.892 INFO:teuthology.orchestra.run.smithi099.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 865 kB/s | 58 kB 00:00 2024-03-25T16:39:42.925 INFO:teuthology.orchestra.run.smithi099.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.5 MB/s | 51 kB 00:00 2024-03-25T16:39:42.950 INFO:teuthology.orchestra.run.smithi099.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.8 MB/s | 548 kB 00:00 2024-03-25T16:39:43.234 INFO:teuthology.orchestra.run.smithi099.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 196 kB/s | 138 kB 00:00 2024-03-25T16:39:43.235 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:43.235 INFO:teuthology.orchestra.run.smithi099.stdout:Total 6.7 MB/s | 6.8 MB 00:01 2024-03-25T16:39:43.248 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:39:43.259 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:39:43.259 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:39:43.346 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:39:43.348 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:39:43.651 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:39:43.815 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:39:43.868 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:39:44.390 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:39:44.470 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:39:44.532 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:39:44.556 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:39:45.398 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:39:45.398 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:39:45.398 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:39:45.399 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:39:45.399 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:39:45.399 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:39:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:39:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:45.706 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:39:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:39:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:39:45.706 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:39:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:39:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:39:45.707 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:45.707 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:39:45.975 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-fuse 2024-03-25T16:39:46.541 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:06:43 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:39:46.690 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:39:46.691 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:39:46.691 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-03-25T16:39:46.691 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:39:46.691 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:39:46.691 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:39:46.691 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-03-25T16:39:46.691 INFO:teuthology.orchestra.run.smithi099.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:39:46.691 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:46.691 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:39:46.692 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:39:46.692 INFO:teuthology.orchestra.run.smithi099.stdout:Install 2 Packages 2024-03-25T16:39:46.692 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:46.692 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 964 k 2024-03-25T16:39:46.692 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 2.8 M 2024-03-25T16:39:46.692 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:39:46.877 INFO:teuthology.orchestra.run.smithi099.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 428 kB/s | 79 kB 00:00 2024-03-25T16:39:49.526 INFO:teuthology.orchestra.run.smithi099.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 312 kB/s | 885 kB 00:02 2024-03-25T16:39:49.528 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:49.528 INFO:teuthology.orchestra.run.smithi099.stdout:Total 340 kB/s | 964 kB 00:02 2024-03-25T16:39:49.528 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:39:49.551 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:39:49.551 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:39:49.627 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:39:49.627 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:39:49.820 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:39:49.952 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:39:50.018 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:39:50.689 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:39:50.689 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:39:50.978 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:39:50.978 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:50.978 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:39:50.978 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:39:50.978 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:50.978 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:39:51.257 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-volume 2024-03-25T16:39:51.825 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:06:48 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:39:51.977 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:39:51.977 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:39:51.977 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-03-25T16:39:51.977 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:39:51.977 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:39:51.977 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:39:51.977 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-03-25T16:39:51.978 INFO:teuthology.orchestra.run.smithi099.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:39:51.978 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:51.978 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:39:51.978 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:39:51.978 INFO:teuthology.orchestra.run.smithi099.stdout:Install 2 Packages 2024-03-25T16:39:51.978 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:51.979 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 332 k 2024-03-25T16:39:51.979 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 1.5 M 2024-03-25T16:39:51.979 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:39:52.091 INFO:teuthology.orchestra.run.smithi099.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 615 kB/s | 69 kB 00:00 2024-03-25T16:39:53.416 INFO:teuthology.orchestra.run.smithi099.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 184 kB/s | 264 kB 00:01 2024-03-25T16:39:53.417 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:53.418 INFO:teuthology.orchestra.run.smithi099.stdout:Total 231 kB/s | 332 kB 00:01 2024-03-25T16:39:53.418 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:39:53.427 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:39:53.427 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:39:53.474 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:39:53.475 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:39:53.625 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:39:53.799 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:39:53.835 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:39:53.850 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:39:53.851 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:39:53.851 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:39:53.851 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:54.562 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:39:54.911 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:39:54.911 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:54.911 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:39:54.911 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:54.912 INFO:teuthology.orchestra.run.smithi099.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:39:54.912 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:54.912 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:39:55.222 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados-devel 2024-03-25T16:39:55.804 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:06:52 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:39:55.952 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:39:55.953 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:39:55.954 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T16:39:55.954 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:39:55.954 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:39:55.954 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:39:55.954 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:55.954 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:39:55.954 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:39:55.954 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T16:39:55.954 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:55.954 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 127 k 2024-03-25T16:39:55.954 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 456 k 2024-03-25T16:39:55.954 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:39:57.054 INFO:teuthology.orchestra.run.smithi099.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 115 kB/s | 127 kB 00:01 2024-03-25T16:39:57.054 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:57.055 INFO:teuthology.orchestra.run.smithi099.stdout:Total 115 kB/s | 127 kB 00:01 2024-03-25T16:39:57.055 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:39:57.062 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:39:57.062 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:39:57.123 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:39:57.123 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:39:57.304 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:39:57.388 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:57.861 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:58.134 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:58.134 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:58.134 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:39:58.134 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:58.134 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:58.135 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:39:58.389 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs2 2024-03-25T16:39:58.953 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:06:55 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:39:59.052 INFO:teuthology.orchestra.run.smithi099.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:59.106 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:39:59.107 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-03-25T16:39:59.107 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:39:59.165 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs-devel 2024-03-25T16:39:59.735 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:06:56 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:39:59.884 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:39:59.885 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:39:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T16:39:59.885 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:39:59.885 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:39:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:39:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:59.885 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:39:59.885 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:39:59.885 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T16:39:59.885 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:39:59.885 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 31 k 2024-03-25T16:39:59.886 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 140 k 2024-03-25T16:39:59.886 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:40:00.498 INFO:teuthology.orchestra.run.smithi099.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 51 kB/s | 31 kB 00:00 2024-03-25T16:40:00.499 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:40:00.499 INFO:teuthology.orchestra.run.smithi099.stdout:Total 51 kB/s | 31 kB 00:00 2024-03-25T16:40:00.499 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:40:00.503 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:40:00.503 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:40:00.531 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:40:00.531 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:40:00.652 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:40:00.779 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:01.170 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:01.442 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:01.442 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:01.442 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:40:01.442 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:40:01.442 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:01.442 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:40:01.624 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados2 2024-03-25T16:40:02.189 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:06:59 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:40:02.285 INFO:teuthology.orchestra.run.smithi099.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:40:02.336 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:40:02.337 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-03-25T16:40:02.337 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:40:02.389 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librbd1 2024-03-25T16:40:02.964 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:06:59 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:40:03.071 INFO:teuthology.orchestra.run.smithi099.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:40:03.127 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:40:03.128 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-03-25T16:40:03.128 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:40:03.186 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rados 2024-03-25T16:40:03.766 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:07:00 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:40:03.864 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:40:03.915 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:40:03.916 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-03-25T16:40:03.916 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:40:03.964 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rgw 2024-03-25T16:40:04.528 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:07:01 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:40:04.623 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:40:04.674 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:40:04.675 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-03-25T16:40:04.675 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:40:04.730 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-cephfs 2024-03-25T16:40:05.295 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:07:02 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:40:05.391 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:40:05.442 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:40:05.443 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-03-25T16:40:05.443 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:40:05.491 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rbd 2024-03-25T16:40:06.053 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:07:03 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:40:06.148 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:40:06.201 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:40:06.202 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-03-25T16:40:06.202 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:40:06.257 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-fuse 2024-03-25T16:40:06.822 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:07:03 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:40:06.970 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:40:06.971 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:40:06.971 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-03-25T16:40:06.971 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:40:06.971 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:40:06.971 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:40:06.971 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:06.972 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:40:06.972 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:40:06.972 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T16:40:06.972 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:06.972 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 85 k 2024-03-25T16:40:06.972 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 231 k 2024-03-25T16:40:06.972 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:40:07.776 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 106 kB/s | 85 kB 00:00 2024-03-25T16:40:07.776 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:40:07.776 INFO:teuthology.orchestra.run.smithi099.stdout:Total 106 kB/s | 85 kB 00:00 2024-03-25T16:40:07.777 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:40:07.784 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:40:07.784 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:40:07.840 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:40:07.840 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:40:07.978 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:40:08.083 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:08.488 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:08.792 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:08.792 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:08.792 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:40:08.792 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:40:08.792 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:08.792 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:40:09.004 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-mirror 2024-03-25T16:40:09.575 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:07:06 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:40:09.729 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:40:09.730 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:40:09.730 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T16:40:09.730 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:40:09.730 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:40:09.730 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:40:09.730 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:09.730 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:40:09.731 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:40:09.731 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T16:40:09.731 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:09.731 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 3.1 M 2024-03-25T16:40:09.731 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 13 M 2024-03-25T16:40:09.732 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:40:14.433 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 260 kB/s | 22 MB 01:28 2024-03-25T16:40:18.752 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 350 kB/s | 3.1 MB 00:09 2024-03-25T16:40:18.753 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:40:18.754 INFO:teuthology.orchestra.run.smithi099.stdout:Total 350 kB/s | 3.1 MB 00:09 2024-03-25T16:40:18.754 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:40:18.764 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:40:18.765 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:40:18.841 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:40:18.841 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:40:19.271 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:40:19.344 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:40:19.368 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:40:19.369 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:20.455 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:20.456 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:20.456 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:40:20.456 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:40:20.456 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:20.456 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:40:20.707 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-nbd 2024-03-25T16:40:21.284 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:07:18 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:40:21.433 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:40:21.434 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:40:21.434 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-03-25T16:40:21.434 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:40:21.434 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:40:21.434 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:40:21.434 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:21.434 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:40:21.434 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:40:21.434 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T16:40:21.434 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:21.434 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 171 k 2024-03-25T16:40:21.435 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 494 k 2024-03-25T16:40:21.435 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:40:23.296 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 92 kB/s | 171 kB 00:01 2024-03-25T16:40:23.297 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:40:23.297 INFO:teuthology.orchestra.run.smithi099.stdout:Total 92 kB/s | 171 kB 00:01 2024-03-25T16:40:23.297 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:40:23.306 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:40:23.306 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:40:23.365 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:40:23.365 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:40:23.522 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:40:23.628 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:24.062 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:24.356 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:24.357 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:24.357 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:40:24.357 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:40:24.357 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:24.357 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:40:24.572 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:40:25.147 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:07:22 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:40:25.299 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:40:25.300 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:40:25.300 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T16:40:25.300 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:40:25.300 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:40:25.301 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:40:25.301 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-03-25T16:40:25.301 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:40:25.301 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:25.301 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:40:25.301 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:40:25.301 INFO:teuthology.orchestra.run.smithi099.stdout:Install 2 Packages 2024-03-25T16:40:25.301 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:40:25.301 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 91 M 2024-03-25T16:40:25.301 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 498 M 2024-03-25T16:40:25.301 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T16:41:24.829 INFO:teuthology.orchestra.run.smithi099.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 386 kB/s | 22 MB 00:59 2024-03-25T16:43:50.723 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 229 kB/s | 68 MB 05:04 2024-03-25T16:43:50.724 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:43:50.724 INFO:teuthology.orchestra.run.smithi190.stdout:Total 304 kB/s | 91 MB 05:04 2024-03-25T16:43:50.724 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T16:43:50.728 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T16:43:50.728 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T16:43:51.317 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T16:43:51.317 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T16:43:54.402 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T16:44:01.534 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:44:01.673 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:44:05.034 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:44:05.034 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:44:05.310 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:44:05.310 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:44:05.311 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T16:44:05.311 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:44:05.311 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:44:05.311 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:44:05.311 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T16:44:05.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:44:06.104 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:11:02 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T16:44:06.260 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T16:44:06.261 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:44:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T16:44:06.261 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:44:06.261 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T16:44:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:44:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:44:06.261 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T16:44:06.261 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T16:44:06.261 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T16:44:06.261 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T16:44:06.261 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 416 M 2024-03-25T16:44:06.261 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 1.7 G 2024-03-25T16:44:06.261 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T16:44:59.702 INFO:teuthology.orchestra.run.smithi099.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 254 kB/s | 68 MB 04:34 2024-03-25T16:44:59.703 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:44:59.703 INFO:teuthology.orchestra.run.smithi099.stdout:Total 338 kB/s | 91 MB 04:34 2024-03-25T16:44:59.704 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T16:44:59.708 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T16:44:59.708 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T16:45:00.299 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T16:45:00.299 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T16:45:03.392 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T16:45:10.575 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:45:10.893 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:45:14.288 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:45:14.288 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:45:14.539 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:45:14.540 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:45:14.540 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T16:45:14.540 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:45:14.540 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:45:14.540 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:45:14.540 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T16:45:14.782 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:45:15.368 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:12:12 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T16:45:15.524 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T16:45:15.525 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:45:15.525 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T16:45:15.525 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:45:15.525 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T16:45:15.525 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:45:15.525 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:45:15.525 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T16:45:15.525 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T16:45:15.525 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T16:45:15.525 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T16:45:15.526 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 416 M 2024-03-25T16:45:15.526 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 1.7 G 2024-03-25T16:45:15.526 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T17:04:33.430 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 347 kB/s | 416 MB 20:27 2024-03-25T17:04:33.430 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:04:33.430 INFO:teuthology.orchestra.run.smithi190.stdout:Total 347 kB/s | 416 MB 20:27 2024-03-25T17:04:33.430 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T17:04:33.434 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T17:04:33.434 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T17:04:35.901 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T17:04:35.901 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T17:04:43.692 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 365 kB/s | 416 MB 19:28 2024-03-25T17:04:43.692 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:04:43.693 INFO:teuthology.orchestra.run.smithi099.stdout:Total 365 kB/s | 416 MB 19:28 2024-03-25T17:04:43.693 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T17:04:43.697 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T17:04:43.697 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T17:04:46.171 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T17:04:46.172 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T17:05:18.095 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T17:05:18.189 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:05:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:05:26.350 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:05:26.350 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:05:26.350 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T17:05:26.350 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:05:26.350 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:05:26.350 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T17:05:26.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:05:27.173 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:32:23 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T17:05:27.321 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T17:05:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-25T17:05:27.322 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T17:05:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-25T17:05:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T17:05:27.322 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:05:27.322 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:05:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T17:05:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-25T17:05:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T17:05:27.323 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:05:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 1.4 M 2024-03-25T17:05:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 6.4 M 2024-03-25T17:05:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T17:05:28.604 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T17:05:28.819 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:05:30.802 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 420 kB/s | 1.4 MB 00:03 2024-03-25T17:05:30.803 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:05:30.803 INFO:teuthology.orchestra.run.smithi190.stdout:Total 420 kB/s | 1.4 MB 00:03 2024-03-25T17:05:30.803 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T17:05:30.807 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T17:05:30.807 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T17:05:30.852 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T17:05:30.852 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T17:05:31.118 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T17:05:31.234 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:05:31.610 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:05:31.880 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:05:31.880 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:05:31.880 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T17:05:31.880 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:05:31.880 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:05:31.881 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T17:05:32.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:05:32.526 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:05:32.622 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:32:28 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T17:05:32.772 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T17:05:32.773 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:05:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T17:05:32.773 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:05:32.773 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T17:05:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:05:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:05:32.773 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T17:05:32.773 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:05:32.773 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T17:05:32.773 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:05:32.773 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 176 M 2024-03-25T17:05:32.773 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 753 M 2024-03-25T17:05:32.773 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T17:05:32.794 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:05:32.794 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:05:32.794 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T17:05:32.794 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:05:32.794 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:05:32.794 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T17:05:33.048 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:05:33.610 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:32:30 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T17:05:33.763 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T17:05:33.764 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2024-03-25T17:05:33.764 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T17:05:33.764 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2024-03-25T17:05:33.764 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T17:05:33.764 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:05:33.764 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:05:33.764 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T17:05:33.764 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2024-03-25T17:05:33.764 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T17:05:33.764 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:05:33.764 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 1.4 M 2024-03-25T17:05:33.764 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 6.4 M 2024-03-25T17:05:33.765 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T17:05:39.536 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 253 kB/s | 1.4 MB 00:05 2024-03-25T17:05:39.537 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:05:39.537 INFO:teuthology.orchestra.run.smithi099.stdout:Total 253 kB/s | 1.4 MB 00:05 2024-03-25T17:05:39.538 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T17:05:39.542 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T17:05:39.542 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T17:05:39.586 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T17:05:39.586 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T17:05:39.840 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T17:05:39.952 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:05:40.354 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:05:40.632 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:05:40.632 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:05:40.632 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T17:05:40.632 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:05:40.632 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:05:40.632 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T17:05:40.829 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:05:41.390 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:32:38 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T17:05:41.538 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T17:05:41.539 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:05:41.539 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T17:05:41.539 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:05:41.539 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T17:05:41.539 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:05:41.539 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:05:41.539 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T17:05:41.540 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:05:41.540 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T17:05:41.540 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:05:41.540 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 176 M 2024-03-25T17:05:41.540 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 753 M 2024-03-25T17:05:41.540 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T17:13:07.725 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 403 kB/s | 176 MB 07:26 2024-03-25T17:13:07.725 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:13:07.725 INFO:teuthology.orchestra.run.smithi099.stdout:Total 403 kB/s | 176 MB 07:26 2024-03-25T17:13:07.725 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T17:13:07.728 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T17:13:07.728 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T17:13:08.788 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T17:13:08.788 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T17:13:26.800 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T17:13:26.885 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:13:31.658 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:13:31.933 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:13:31.933 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:13:31.933 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T17:13:31.933 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:13:31.933 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:13:31.933 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T17:13:32.167 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:13:32.729 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:40:29 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T17:13:32.879 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T17:13:32.879 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:13:32.879 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T17:13:32.879 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:13:32.879 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T17:13:32.879 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:13:32.880 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:13:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T17:13:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:13:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T17:13:32.880 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:13:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 672 M 2024-03-25T17:13:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 3.1 G 2024-03-25T17:13:32.880 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T17:15:46.124 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 293 kB/s | 176 MB 10:13 2024-03-25T17:15:46.124 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:46.124 INFO:teuthology.orchestra.run.smithi190.stdout:Total 293 kB/s | 176 MB 10:13 2024-03-25T17:15:46.124 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T17:15:46.127 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T17:15:46.127 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T17:15:47.185 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T17:15:47.185 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T17:16:05.359 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T17:16:05.468 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:16:10.003 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:16:10.274 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:16:10.274 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:16:10.274 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T17:16:10.275 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:16:10.275 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:16:10.275 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T17:16:10.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:16:11.053 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:43:07 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T17:16:11.214 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T17:16:11.215 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:16:11.215 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T17:16:11.216 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:16:11.216 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T17:16:11.216 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:16:11.216 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:16:11.216 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T17:16:11.216 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:16:11.216 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T17:16:11.216 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:16:11.216 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 672 M 2024-03-25T17:16:11.216 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 3.1 G 2024-03-25T17:16:11.216 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T17:48:28.976 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 328 kB/s | 672 MB 34:56 2024-03-25T17:48:28.998 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:48:28.998 INFO:teuthology.orchestra.run.smithi099.stdout:Total 328 kB/s | 672 MB 34:56 2024-03-25T17:48:28.998 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T17:48:28.999 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T17:48:28.999 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T17:48:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T17:48:32.937 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T17:49:09.155 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 348 kB/s | 672 MB 32:57 2024-03-25T17:49:09.155 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:09.156 INFO:teuthology.orchestra.run.smithi190.stdout:Total 348 kB/s | 672 MB 32:57 2024-03-25T17:49:09.156 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T17:49:09.161 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T17:49:09.161 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T17:49:13.126 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T17:49:13.126 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T17:49:43.899 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T17:49:44.069 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:49:53.265 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:49:53.535 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:49:53.535 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:49:53.535 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T17:49:53.535 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:53.535 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:49:53.535 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T17:49:53.859 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:49:54.427 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:16:51 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T17:49:54.524 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:49:54.575 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T17:49:54.576 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-03-25T17:49:54.576 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T17:49:54.643 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:49:55.206 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:16:52 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T17:49:55.353 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T17:49:55.354 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:49:55.354 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T17:49:55.354 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:49:55.354 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T17:49:55.354 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:49:55.355 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:49:55.355 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T17:49:55.355 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:49:55.355 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T17:49:55.355 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:49:55.355 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 20 M 2024-03-25T17:49:55.355 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 103 M 2024-03-25T17:49:55.355 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T17:50:24.131 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T17:50:24.691 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:50:33.540 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:50:33.835 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:50:33.835 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:50:33.835 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T17:50:33.835 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:33.835 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:50:33.835 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T17:50:34.103 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:50:34.672 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:17:30 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T17:50:34.768 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:50:34.820 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T17:50:34.821 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-25T17:50:34.821 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T17:50:34.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:17:31 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T17:50:35.616 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T17:50:35.617 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:50:35.617 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T17:50:35.617 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:50:35.617 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T17:50:35.617 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:50:35.618 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:50:35.618 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T17:50:35.618 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:50:35.618 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T17:50:35.618 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:50:35.618 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 20 M 2024-03-25T17:50:35.618 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 103 M 2024-03-25T17:50:35.618 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T17:50:49.870 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 370 kB/s | 20 MB 00:54 2024-03-25T17:50:49.870 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:50:49.871 INFO:teuthology.orchestra.run.smithi099.stdout:Total 370 kB/s | 20 MB 00:54 2024-03-25T17:50:49.871 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T17:50:49.875 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T17:50:49.875 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T17:50:50.026 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T17:50:50.026 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T17:50:52.260 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T17:50:52.567 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:50:53.539 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:50:53.804 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:50:53.804 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:50:53.804 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T17:50:53.804 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:53.804 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:50:53.804 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T17:50:54.036 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:50:54.603 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:17:51 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T17:50:54.750 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T17:50:54.751 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:50:54.751 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T17:50:54.751 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:50:54.751 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T17:50:54.751 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:50:54.751 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:50:54.751 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T17:50:54.751 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:50:54.751 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T17:50:54.751 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:50:54.751 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 48 M 2024-03-25T17:50:54.752 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 173 M 2024-03-25T17:50:54.752 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T17:51:22.453 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 431 kB/s | 20 MB 00:46 2024-03-25T17:51:22.454 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:22.454 INFO:teuthology.orchestra.run.smithi190.stdout:Total 431 kB/s | 20 MB 00:46 2024-03-25T17:51:22.455 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T17:51:22.458 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T17:51:22.459 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T17:51:22.609 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T17:51:22.609 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T17:51:24.840 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T17:51:24.934 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:25.837 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:51:26.101 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T17:51:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:51:26.101 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T17:51:26.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:51:26.826 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:18:22 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T17:51:26.974 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T17:51:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:51:26.975 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T17:51:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:51:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T17:51:26.975 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:51:26.975 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:51:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T17:51:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:51:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T17:51:26.975 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:51:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 48 M 2024-03-25T17:51:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 173 M 2024-03-25T17:51:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T17:53:47.262 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 283 kB/s | 48 MB 02:52 2024-03-25T17:53:47.263 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:47.264 INFO:teuthology.orchestra.run.smithi099.stdout:Total 283 kB/s | 48 MB 02:52 2024-03-25T17:53:47.264 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T17:53:47.266 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T17:53:47.266 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T17:53:47.579 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T17:53:47.580 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T17:53:52.325 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T17:53:52.426 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:53.817 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:54.097 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:54.098 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:53:54.098 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T17:53:54.098 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:54.098 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:53:54.098 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T17:53:54.294 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:53:54.858 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:20:51 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T17:53:55.006 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T17:53:55.006 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:53:55.007 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T17:53:55.007 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:53:55.007 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T17:53:55.007 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:53:55.007 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:53:55.007 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T17:53:55.007 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:53:55.007 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T17:53:55.007 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:53:55.007 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 99 M 2024-03-25T17:53:55.007 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 449 M 2024-03-25T17:53:55.007 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T17:54:12.159 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 296 kB/s | 48 MB 02:45 2024-03-25T17:54:12.159 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:12.159 INFO:teuthology.orchestra.run.smithi190.stdout:Total 296 kB/s | 48 MB 02:45 2024-03-25T17:54:12.160 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T17:54:12.163 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T17:54:12.163 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T17:54:12.476 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T17:54:12.477 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T17:54:17.212 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T17:54:17.332 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:18.676 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:18.949 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:18.950 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:54:18.950 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T17:54:18.950 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:18.950 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:54:18.950 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T17:54:19.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:54:19.676 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:21:15 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T17:54:19.825 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T17:54:19.826 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:54:19.826 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T17:54:19.826 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:54:19.826 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T17:54:19.826 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:54:19.826 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:54:19.826 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T17:54:19.827 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:54:19.827 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T17:54:19.827 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:54:19.827 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 99 M 2024-03-25T17:54:19.827 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 449 M 2024-03-25T17:54:19.827 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T17:58:06.916 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 444 kB/s | 99 MB 03:47 2024-03-25T17:58:06.916 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:06.916 INFO:teuthology.orchestra.run.smithi190.stdout:Total 444 kB/s | 99 MB 03:47 2024-03-25T17:58:06.916 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T17:58:06.919 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T17:58:06.919 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T17:58:07.526 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T17:58:07.526 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T17:58:17.819 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T17:58:17.920 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:20.739 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:21.044 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:21.044 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:58:21.045 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T17:58:21.045 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:21.045 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:58:21.045 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T17:58:21.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:58:21.761 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:25:17 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T17:58:21.909 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T17:58:21.910 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:58:21.910 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T17:58:21.910 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:58:21.911 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T17:58:21.911 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:58:21.911 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:58:21.911 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T17:58:21.911 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T17:58:21.911 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T17:58:21.911 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T17:58:21.911 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 287 M 2024-03-25T17:58:21.911 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 1.2 G 2024-03-25T17:58:21.911 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T17:58:48.554 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 344 kB/s | 99 MB 04:53 2024-03-25T17:58:48.555 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:48.555 INFO:teuthology.orchestra.run.smithi099.stdout:Total 344 kB/s | 99 MB 04:53 2024-03-25T17:58:48.555 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T17:58:48.558 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T17:58:48.558 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T17:58:49.169 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T17:58:49.169 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T17:58:59.442 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T17:58:59.542 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:59:02.479 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:59:02.753 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:59:02.753 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:59:02.753 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T17:59:02.753 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:02.753 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:59:02.754 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T17:59:02.968 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:59:03.535 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:26:00 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T17:59:03.684 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T17:59:03.685 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:59:03.685 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T17:59:03.685 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:59:03.685 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T17:59:03.685 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:59:03.685 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:59:03.685 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T17:59:03.685 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T17:59:03.685 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T17:59:03.685 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T17:59:03.685 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 287 M 2024-03-25T17:59:03.685 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 1.2 G 2024-03-25T17:59:03.685 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T18:15:30.487 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 286 kB/s | 287 MB 17:08 2024-03-25T18:15:30.488 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:30.488 INFO:teuthology.orchestra.run.smithi190.stdout:Total 286 kB/s | 287 MB 17:08 2024-03-25T18:15:30.488 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T18:15:30.491 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T18:15:30.491 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T18:15:32.204 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T18:15:32.204 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T18:16:01.735 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T18:16:02.277 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:04.686 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:04.995 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:04.995 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:04.995 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T18:16:04.996 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:04.996 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:04.996 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T18:16:05.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:16:05.775 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:43:01 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T18:16:05.933 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T18:16:05.934 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:16:05.934 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T18:16:05.934 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:16:05.934 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T18:16:05.934 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:16:05.934 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:05.934 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T18:16:05.934 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:16:05.934 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T18:16:05.935 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:05.935 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 5.9 M 2024-03-25T18:16:05.935 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 28 M 2024-03-25T18:16:05.935 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T18:16:15.694 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 624 kB/s | 5.9 MB 00:09 2024-03-25T18:16:15.695 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:15.695 INFO:teuthology.orchestra.run.smithi190.stdout:Total 623 kB/s | 5.9 MB 00:09 2024-03-25T18:16:15.695 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T18:16:15.699 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T18:16:15.699 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T18:16:15.772 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T18:16:15.772 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T18:16:16.511 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T18:16:16.626 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:16:17.198 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:16:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:16:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:17.468 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T18:16:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:17.468 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:17.468 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T18:16:17.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:16:18.191 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:43:14 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T18:16:18.341 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T18:16:18.342 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:16:18.342 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T18:16:18.342 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:16:18.342 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T18:16:18.342 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:16:18.342 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:18.342 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T18:16:18.342 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:16:18.342 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T18:16:18.342 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:18.343 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 109 k 2024-03-25T18:16:18.343 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 440 k 2024-03-25T18:16:18.343 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T18:16:19.207 INFO:teuthology.orchestra.run.smithi190.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 126 kB/s | 109 kB 00:00 2024-03-25T18:16:19.208 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:19.208 INFO:teuthology.orchestra.run.smithi190.stdout:Total 126 kB/s | 109 kB 00:00 2024-03-25T18:16:19.208 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T18:16:19.212 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T18:16:19.212 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T18:16:19.247 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T18:16:19.248 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T18:16:19.368 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T18:16:19.472 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:16:19.889 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:16:20.169 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:16:20.169 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:20.169 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T18:16:20.169 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:20.169 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:20.169 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T18:16:20.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:16:20.901 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:43:16 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T18:16:21.049 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T18:16:21.050 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:16:21.050 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T18:16:21.050 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:16:21.050 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T18:16:21.050 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:16:21.050 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:21.050 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T18:16:21.051 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:16:21.051 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T18:16:21.051 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:21.051 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 5.4 M 2024-03-25T18:16:21.051 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 25 M 2024-03-25T18:16:21.051 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T18:16:33.142 INFO:teuthology.orchestra.run.smithi190.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 453 kB/s | 5.4 MB 00:12 2024-03-25T18:16:33.143 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:33.143 INFO:teuthology.orchestra.run.smithi190.stdout:Total 453 kB/s | 5.4 MB 00:12 2024-03-25T18:16:33.144 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T18:16:33.148 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T18:16:33.148 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T18:16:33.214 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T18:16:33.214 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T18:16:33.883 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T18:16:33.990 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:16:34.492 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:16:34.757 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:16:34.757 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:34.757 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T18:16:34.757 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:34.757 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:34.757 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T18:16:34.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados2-debuginfo 2024-03-25T18:16:35.500 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:43:31 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T18:16:35.651 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T18:16:35.652 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:16:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T18:16:35.652 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:16:35.652 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T18:16:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:16:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:35.652 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T18:16:35.652 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:16:35.652 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T18:16:35.652 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:16:35.652 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 44 M 2024-03-25T18:16:35.652 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 253 M 2024-03-25T18:16:35.652 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T18:16:48.204 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 276 kB/s | 287 MB 17:44 2024-03-25T18:16:48.205 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:48.205 INFO:teuthology.orchestra.run.smithi099.stdout:Total 276 kB/s | 287 MB 17:44 2024-03-25T18:16:48.205 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T18:16:48.209 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T18:16:48.209 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T18:16:49.924 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T18:16:49.924 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T18:17:19.611 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T18:17:20.176 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:17:22.446 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:17:22.735 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:17:22.735 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:17:22.735 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T18:17:22.735 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:22.735 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:17:22.736 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T18:17:22.955 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:17:23.521 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:44:20 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T18:17:23.670 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T18:17:23.670 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:17:23.670 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T18:17:23.670 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:17:23.671 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T18:17:23.671 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:17:23.671 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:17:23.671 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T18:17:23.671 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:17:23.671 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T18:17:23.671 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:17:23.672 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 5.9 M 2024-03-25T18:17:23.672 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 28 M 2024-03-25T18:17:23.672 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T18:17:41.832 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 335 kB/s | 5.9 MB 00:18 2024-03-25T18:17:41.832 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:41.833 INFO:teuthology.orchestra.run.smithi099.stdout:Total 335 kB/s | 5.9 MB 00:18 2024-03-25T18:17:41.833 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T18:17:41.837 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T18:17:41.837 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T18:17:41.908 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T18:17:41.908 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T18:17:42.638 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T18:17:42.738 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:17:43.316 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:17:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:17:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:17:43.600 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T18:17:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:43.600 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:17:43.600 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T18:17:43.808 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:17:44.368 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:44:41 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T18:17:44.518 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T18:17:44.519 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:17:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T18:17:44.519 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:17:44.519 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T18:17:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:17:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:17:44.519 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T18:17:44.519 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:17:44.519 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T18:17:44.519 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:17:44.519 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 109 k 2024-03-25T18:17:44.519 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 440 k 2024-03-25T18:17:44.519 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T18:17:45.159 INFO:teuthology.orchestra.run.smithi099.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 170 kB/s | 109 kB 00:00 2024-03-25T18:17:45.159 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:45.160 INFO:teuthology.orchestra.run.smithi099.stdout:Total 170 kB/s | 109 kB 00:00 2024-03-25T18:17:45.160 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T18:17:45.163 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T18:17:45.164 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T18:17:45.199 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T18:17:45.199 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T18:17:45.317 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T18:17:45.433 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:17:45.815 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:17:46.092 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:17:46.092 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:17:46.092 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T18:17:46.093 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:46.093 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:17:46.093 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T18:17:46.301 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:17:46.859 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:44:43 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T18:17:47.007 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T18:17:47.007 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:17:47.007 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T18:17:47.007 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:17:47.008 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T18:17:47.008 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:17:47.008 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:17:47.008 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T18:17:47.008 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:17:47.008 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T18:17:47.008 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:17:47.008 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 5.4 M 2024-03-25T18:17:47.009 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 25 M 2024-03-25T18:17:47.009 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T18:18:04.850 INFO:teuthology.orchestra.run.smithi099.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 307 kB/s | 5.4 MB 00:17 2024-03-25T18:18:04.851 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:04.851 INFO:teuthology.orchestra.run.smithi099.stdout:Total 307 kB/s | 5.4 MB 00:17 2024-03-25T18:18:04.852 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T18:18:04.856 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T18:18:04.856 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T18:18:04.923 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T18:18:04.924 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T18:18:05.612 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T18:18:05.723 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:06.270 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:06.546 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:06.546 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:18:06.547 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T18:18:06.547 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:06.547 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:18:06.547 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T18:18:06.767 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados2-debuginfo 2024-03-25T18:18:07.332 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:45:04 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T18:18:07.480 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T18:18:07.480 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:18:07.480 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T18:18:07.480 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:18:07.481 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T18:18:07.481 INFO:teuthology.orchestra.run.smithi099.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:18:07.481 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:18:07.481 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T18:18:07.481 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:18:07.481 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T18:18:07.481 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:18:07.481 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 44 M 2024-03-25T18:18:07.481 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 253 M 2024-03-25T18:18:07.482 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T18:18:12.615 INFO:teuthology.orchestra.run.smithi190.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 468 kB/s | 44 MB 01:36 2024-03-25T18:18:12.616 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:12.616 INFO:teuthology.orchestra.run.smithi190.stdout:Total 468 kB/s | 44 MB 01:36 2024-03-25T18:18:12.616 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T18:18:12.620 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T18:18:12.620 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T18:18:12.919 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T18:18:12.919 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T18:18:17.952 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T18:18:18.122 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:19.840 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:20.107 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:20.108 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:18:20.108 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T18:18:20.108 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:20.108 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:18:20.108 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T18:18:20.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:18:20.844 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:45:16 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T18:18:20.995 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T18:18:20.995 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:18:20.995 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T18:18:20.996 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:18:20.996 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T18:18:20.996 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:18:20.996 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:18:20.996 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T18:18:20.996 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:18:20.996 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T18:18:20.996 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:18:20.997 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 52 M 2024-03-25T18:18:20.997 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 308 M 2024-03-25T18:18:20.997 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T18:20:01.838 INFO:teuthology.orchestra.run.smithi099.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 397 kB/s | 44 MB 01:54 2024-03-25T18:20:01.839 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:01.839 INFO:teuthology.orchestra.run.smithi099.stdout:Total 397 kB/s | 44 MB 01:54 2024-03-25T18:20:01.839 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T18:20:01.843 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T18:20:01.843 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T18:20:02.140 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T18:20:02.140 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T18:20:07.171 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T18:20:07.269 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:20:08.461 INFO:teuthology.orchestra.run.smithi190.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 494 kB/s | 52 MB 01:47 2024-03-25T18:20:08.463 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:08.463 INFO:teuthology.orchestra.run.smithi190.stdout:Total 494 kB/s | 52 MB 01:47 2024-03-25T18:20:08.463 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T18:20:08.466 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T18:20:08.466 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T18:20:08.813 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T18:20:08.813 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T18:20:09.080 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:20:09.377 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:20:09.377 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:20:09.377 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T18:20:09.377 INFO:teuthology.orchestra.run.smithi099.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:09.377 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:20:09.377 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T18:20:09.566 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:20:10.129 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:47:07 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T18:20:10.279 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T18:20:10.279 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:20:10.280 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T18:20:10.280 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:20:10.280 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T18:20:10.280 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:20:10.280 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:20:10.280 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T18:20:10.280 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:20:10.280 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T18:20:10.280 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:20:10.281 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 52 M 2024-03-25T18:20:10.281 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 308 M 2024-03-25T18:20:10.281 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T18:20:14.784 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T18:20:14.893 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:20:17.038 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:20:17.312 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:20:17.313 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:17.313 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T18:20:17.313 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:17.313 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:17.313 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T18:20:17.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:20:18.061 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:47:14 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T18:20:18.210 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T18:20:18.211 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:18.211 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T18:20:18.211 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:18.211 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T18:20:18.211 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:20:18.211 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:18.212 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T18:20:18.212 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:18.212 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T18:20:18.212 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:18.212 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 615 k 2024-03-25T18:20:18.212 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 2.1 M 2024-03-25T18:20:18.212 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T18:20:20.031 INFO:teuthology.orchestra.run.smithi190.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 338 kB/s | 615 kB 00:01 2024-03-25T18:20:20.031 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:20.032 INFO:teuthology.orchestra.run.smithi190.stdout:Total 338 kB/s | 615 kB 00:01 2024-03-25T18:20:20.032 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T18:20:20.036 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T18:20:20.036 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T18:20:20.077 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T18:20:20.077 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T18:20:20.246 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T18:20:20.346 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:20:20.721 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:20:20.988 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:20:20.988 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:20.988 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T18:20:20.988 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:20.988 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:20.988 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T18:20:21.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:20:21.719 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:47:17 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T18:20:21.870 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T18:20:21.870 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T18:20:21.870 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:21.870 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T18:20:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:20:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:21.870 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T18:20:21.871 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:21.871 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T18:20:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:21.871 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 771 k 2024-03-25T18:20:21.871 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 2.7 M 2024-03-25T18:20:21.871 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T18:20:23.783 INFO:teuthology.orchestra.run.smithi190.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 403 kB/s | 771 kB 00:01 2024-03-25T18:20:23.784 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:23.784 INFO:teuthology.orchestra.run.smithi190.stdout:Total 403 kB/s | 771 kB 00:01 2024-03-25T18:20:23.784 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T18:20:23.788 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T18:20:23.788 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T18:20:23.831 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T18:20:23.831 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T18:20:24.039 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T18:20:24.149 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:20:24.551 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:20:24.839 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:20:24.839 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:24.839 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T18:20:24.839 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:24.839 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:24.839 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T18:20:25.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:20:25.603 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:47:21 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T18:20:25.752 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T18:20:25.753 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:25.753 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T18:20:25.753 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:25.753 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T18:20:25.753 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:20:25.753 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:25.753 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T18:20:25.753 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:25.753 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T18:20:25.753 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:25.754 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 1.1 M 2024-03-25T18:20:25.754 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 4.3 M 2024-03-25T18:20:25.754 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T18:20:28.120 INFO:teuthology.orchestra.run.smithi190.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 486 kB/s | 1.1 MB 00:02 2024-03-25T18:20:28.121 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:28.121 INFO:teuthology.orchestra.run.smithi190.stdout:Total 486 kB/s | 1.1 MB 00:02 2024-03-25T18:20:28.121 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T18:20:28.125 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T18:20:28.125 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T18:20:28.169 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T18:20:28.169 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T18:20:28.410 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T18:20:28.520 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:28.931 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:29.204 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:29.204 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:29.204 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T18:20:29.204 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:29.204 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:29.204 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T18:20:29.384 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:20:29.945 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:47:25 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T18:20:30.099 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T18:20:30.100 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T18:20:30.100 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:30.100 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T18:20:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:20:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:30.100 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T18:20:30.100 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:30.100 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T18:20:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:30.101 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 337 k 2024-03-25T18:20:30.101 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 1.0 M 2024-03-25T18:20:30.101 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T18:20:31.589 INFO:teuthology.orchestra.run.smithi190.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 227 kB/s | 337 kB 00:01 2024-03-25T18:20:31.589 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:31.590 INFO:teuthology.orchestra.run.smithi190.stdout:Total 226 kB/s | 337 kB 00:01 2024-03-25T18:20:31.590 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T18:20:31.594 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T18:20:31.594 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T18:20:31.635 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T18:20:31.635 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T18:20:31.778 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T18:20:31.888 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:32.269 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:32.546 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:32.546 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:32.547 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T18:20:32.547 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:32.547 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:32.547 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T18:20:32.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:20:33.302 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:47:29 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T18:20:33.451 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T18:20:33.452 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:33.452 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T18:20:33.452 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:33.452 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T18:20:33.452 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:20:33.452 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:33.452 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T18:20:33.452 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:33.452 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T18:20:33.453 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:33.453 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 509 k 2024-03-25T18:20:33.453 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 2.5 M 2024-03-25T18:20:33.453 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T18:20:35.541 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 244 kB/s | 509 kB 00:02 2024-03-25T18:20:35.542 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:35.542 INFO:teuthology.orchestra.run.smithi190.stdout:Total 244 kB/s | 509 kB 00:02 2024-03-25T18:20:35.545 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T18:20:35.546 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T18:20:35.547 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T18:20:35.588 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T18:20:35.588 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T18:20:35.749 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T18:20:35.858 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:20:36.267 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:20:36.544 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:20:36.544 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:36.544 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T18:20:36.544 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:36.544 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:36.544 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T18:20:36.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:20:37.274 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:47:33 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T18:20:37.424 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T18:20:37.424 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:37.424 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T18:20:37.424 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:37.424 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T18:20:37.425 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:20:37.425 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:37.425 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T18:20:37.425 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:20:37.425 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T18:20:37.425 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:20:37.426 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 52 M 2024-03-25T18:20:37.426 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 298 M 2024-03-25T18:20:37.426 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T18:21:50.794 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 719 kB/s | 52 MB 01:13 2024-03-25T18:21:50.794 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:50.795 INFO:teuthology.orchestra.run.smithi190.stdout:Total 719 kB/s | 52 MB 01:13 2024-03-25T18:21:50.795 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T18:21:50.798 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T18:21:50.798 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T18:21:51.133 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T18:21:51.133 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T18:21:57.032 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T18:21:57.351 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:21:57.818 INFO:teuthology.orchestra.run.smithi099.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 494 kB/s | 52 MB 01:47 2024-03-25T18:21:57.819 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:57.819 INFO:teuthology.orchestra.run.smithi099.stdout:Total 494 kB/s | 52 MB 01:47 2024-03-25T18:21:57.819 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T18:21:57.823 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T18:21:57.823 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T18:21:58.165 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T18:21:58.166 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T18:21:59.426 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:21:59.705 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:21:59.706 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:21:59.706 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T18:21:59.706 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:59.706 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:21:59.706 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T18:21:59.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:22:00.452 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 1:48:56 ago on Mon 25 Mar 2024 04:33:04 PM UTC. 2024-03-25T18:22:00.601 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-25T18:22:00.602 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:22:00.602 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-25T18:22:00.602 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:22:00.602 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-25T18:22:00.602 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:22:00.602 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:22:00.602 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-25T18:22:00.602 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-25T18:22:00.603 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-25T18:22:00.603 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:22:00.603 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 1.3 M 2024-03-25T18:22:00.603 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 5.5 M 2024-03-25T18:22:00.603 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-25T18:22:02.928 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 551 kB/s | 1.3 MB 00:02 2024-03-25T18:22:02.929 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:02.929 INFO:teuthology.orchestra.run.smithi190.stdout:Total 551 kB/s | 1.3 MB 00:02 2024-03-25T18:22:02.929 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-25T18:22:02.934 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-25T18:22:02.934 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-25T18:22:02.978 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-25T18:22:02.978 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-25T18:22:03.238 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-25T18:22:03.353 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:22:03.761 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:22:04.040 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:22:04.041 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:22:04.041 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-25T18:22:04.041 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:04.041 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:22:04.041 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-25T18:22:04.194 DEBUG:teuthology.parallel:result is None 2024-03-25T18:22:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T18:22:04.299 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:22:06.437 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:22:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:22:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:06.701 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T18:22:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:06.701 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:06.702 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T18:22:06.908 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:22:07.468 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:49:04 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T18:22:07.616 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T18:22:07.617 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:07.617 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T18:22:07.617 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:07.617 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T18:22:07.617 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:22:07.617 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:07.617 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T18:22:07.617 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:07.618 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T18:22:07.618 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:07.618 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 615 k 2024-03-25T18:22:07.618 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 2.1 M 2024-03-25T18:22:07.618 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T18:22:10.303 INFO:teuthology.orchestra.run.smithi099.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 229 kB/s | 615 kB 00:02 2024-03-25T18:22:10.304 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:10.304 INFO:teuthology.orchestra.run.smithi099.stdout:Total 229 kB/s | 615 kB 00:02 2024-03-25T18:22:10.304 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T18:22:10.308 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T18:22:10.308 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T18:22:10.349 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T18:22:10.350 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T18:22:10.517 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T18:22:10.617 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:22:11.046 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:22:11.307 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:22:11.307 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:11.307 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T18:22:11.308 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:11.308 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:11.308 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T18:22:11.550 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:22:12.112 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:49:09 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T18:22:12.262 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T18:22:12.263 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:12.263 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T18:22:12.263 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:12.263 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T18:22:12.263 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:22:12.263 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:12.263 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T18:22:12.263 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:12.263 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T18:22:12.263 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:12.264 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 771 k 2024-03-25T18:22:12.264 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 2.7 M 2024-03-25T18:22:12.264 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T18:22:14.164 INFO:teuthology.orchestra.run.smithi099.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 406 kB/s | 771 kB 00:01 2024-03-25T18:22:14.164 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:14.165 INFO:teuthology.orchestra.run.smithi099.stdout:Total 405 kB/s | 771 kB 00:01 2024-03-25T18:22:14.165 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T18:22:14.169 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T18:22:14.169 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T18:22:14.211 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T18:22:14.211 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T18:22:14.399 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T18:22:14.496 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:22:14.898 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:22:15.165 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:22:15.165 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:15.165 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T18:22:15.166 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:15.166 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:15.166 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T18:22:15.391 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:22:15.953 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:49:12 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T18:22:16.106 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T18:22:16.107 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:16.107 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T18:22:16.107 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:16.107 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T18:22:16.107 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:22:16.107 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:16.107 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T18:22:16.108 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:16.108 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T18:22:16.108 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:16.108 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 1.1 M 2024-03-25T18:22:16.108 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 4.3 M 2024-03-25T18:22:16.108 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T18:22:18.669 INFO:teuthology.orchestra.run.smithi099.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 449 kB/s | 1.1 MB 00:02 2024-03-25T18:22:18.669 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:18.669 INFO:teuthology.orchestra.run.smithi099.stdout:Total 449 kB/s | 1.1 MB 00:02 2024-03-25T18:22:18.670 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T18:22:18.674 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T18:22:18.674 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T18:22:18.716 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T18:22:18.717 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T18:22:18.945 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T18:22:19.051 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:19.449 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:19.723 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:19.724 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:19.724 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T18:22:19.724 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:19.724 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:19.724 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T18:22:19.936 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:22:20.499 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:49:17 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T18:22:20.649 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T18:22:20.649 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:20.650 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T18:22:20.650 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:20.650 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T18:22:20.650 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:22:20.650 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:20.650 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T18:22:20.650 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:20.650 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T18:22:20.650 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:20.650 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 337 k 2024-03-25T18:22:20.650 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 1.0 M 2024-03-25T18:22:20.650 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T18:22:23.038 INFO:teuthology.orchestra.run.smithi099.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 141 kB/s | 337 kB 00:02 2024-03-25T18:22:23.039 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:23.039 INFO:teuthology.orchestra.run.smithi099.stdout:Total 141 kB/s | 337 kB 00:02 2024-03-25T18:22:23.039 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T18:22:23.043 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T18:22:23.043 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T18:22:23.083 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T18:22:23.083 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T18:22:23.221 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T18:22:23.320 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:23.670 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:23.940 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:22:23.940 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:23.940 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T18:22:23.940 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:23.941 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:23.941 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T18:22:24.156 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:22:24.716 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:49:21 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T18:22:24.865 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T18:22:24.866 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:24.866 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T18:22:24.866 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:24.866 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T18:22:24.866 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:22:24.866 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:24.866 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T18:22:24.866 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:24.867 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T18:22:24.867 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:24.867 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 509 k 2024-03-25T18:22:24.867 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 2.5 M 2024-03-25T18:22:24.867 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T18:22:28.024 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 161 kB/s | 509 kB 00:03 2024-03-25T18:22:28.024 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:28.025 INFO:teuthology.orchestra.run.smithi099.stdout:Total 161 kB/s | 509 kB 00:03 2024-03-25T18:22:28.025 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T18:22:28.029 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T18:22:28.029 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T18:22:28.068 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T18:22:28.068 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T18:22:28.234 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T18:22:28.332 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:22:28.700 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:22:28.980 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:22:28.981 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:28.981 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T18:22:28.981 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:22:28.981 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:28.981 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T18:22:29.183 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:22:29.744 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:49:26 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T18:22:29.892 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T18:22:29.893 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:29.893 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T18:22:29.893 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:29.893 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T18:22:29.893 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:22:29.893 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:29.893 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T18:22:29.893 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:22:29.894 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T18:22:29.894 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:22:29.894 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 52 M 2024-03-25T18:22:29.894 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 298 M 2024-03-25T18:22:29.894 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T18:24:09.508 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 530 kB/s | 52 MB 01:39 2024-03-25T18:24:09.508 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:09.509 INFO:teuthology.orchestra.run.smithi099.stdout:Total 530 kB/s | 52 MB 01:39 2024-03-25T18:24:09.509 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T18:24:09.512 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T18:24:09.512 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T18:24:09.848 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T18:24:09.848 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T18:24:15.762 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T18:24:15.872 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:24:17.915 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:24:18.220 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:24:18.220 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:18.221 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T18:24:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:18.221 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:18.221 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T18:24:18.426 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:24:18.992 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 1:51:15 ago on Mon 25 Mar 2024 04:33:03 PM UTC. 2024-03-25T18:24:19.141 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-25T18:24:19.142 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:24:19.142 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2024-03-25T18:24:19.142 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:24:19.142 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-25T18:24:19.142 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:24:19.142 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:19.142 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-25T18:24:19.142 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-25T18:24:19.142 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2024-03-25T18:24:19.142 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:19.142 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 1.3 M 2024-03-25T18:24:19.142 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 5.5 M 2024-03-25T18:24:19.142 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-25T18:24:21.909 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 464 kB/s | 1.3 MB 00:02 2024-03-25T18:24:21.909 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:21.910 INFO:teuthology.orchestra.run.smithi099.stdout:Total 463 kB/s | 1.3 MB 00:02 2024-03-25T18:24:21.910 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-25T18:24:21.914 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-25T18:24:21.914 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-25T18:24:21.957 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-25T18:24:21.957 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-25T18:24:22.191 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-25T18:24:22.299 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:22.701 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:22.969 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:24:22.969 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:22.969 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-25T18:24:22.970 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:22.970 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:22.970 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-25T18:24:23.184 DEBUG:teuthology.parallel:result is None 2024-03-25T18:24:23.184 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:24:23.403 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:24:23.423 INFO:teuthology.orchestra.run.smithi099.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:24:23.424 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:24:23.424 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:24:23.425 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:24:23.640 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:24:23.665 INFO:teuthology.orchestra.run.smithi190.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:24:23.665 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:24:23.665 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:24:23.667 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:24:23.667 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:23.667 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:24:23.694 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:24:23.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:24:23.736 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:24:23.736 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:23.736 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:24:23.763 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:24:23.830 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:24:23.830 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:24:23.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:24:23.932 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:24:23.933 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:23.933 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:24:23.960 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:24:24.027 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:24:24.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:24:24.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:24:24.125 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:24:24.125 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:24.125 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:24:24.152 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:24:24.218 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:24:24.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:24:24.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:24:24.315 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:24:24.399 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:24:24.399 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 777 /var/log/ceph 2024-03-25T18:24:24.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /var/log/ceph 2024-03-25T18:24:24.430 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:24:24.430 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:24:24.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:24:24.497 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:24:24.498 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:24:24.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:24:24.594 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:24:24.594 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:24:24.594 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621278', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi099.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGtxbdLy2UHaHA5gRIf3MR5NU+UzyKuyLZnQy02lhjxuvfdtMPt2OxqNQS24ovTq2LYanLIoZBf0ViH2OnEuAKo=', 'smithi190.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCYd91K16MqN2SbtpXlvonTKb/ApH9+7u27YGHa817euhxeRbawaNy+ucBzzgJtxkgmTJjO1E+7tv7Dyw3xr/rU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:24:24.595 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:24:24.603 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:24:24.620 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:24:24.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:24:24.687 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:24.687 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:24:24.738 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:24:24.738 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_1 2024-03-25T18:24:24.792 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:24:24.792 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:24.792 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T18:24:24.792 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:24.792 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:24.792 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-03-25 16:40:19.622534662 +0000 2024-03-25T18:24:24.792 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-03-25 16:32:19.020767710 +0000 2024-03-25T18:24:24.792 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-03-25 16:32:19.020767710 +0000 2024-03-25T18:24:24.792 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-03-25 16:32:19.020767710 +0000 2024-03-25T18:24:24.792 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:24:24.857 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-03-25T18:24:24.857 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-03-25T18:24:24.857 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000208728 s, 2.5 MB/s 2024-03-25T18:24:24.858 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:24:24.914 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_2 2024-03-25T18:24:24.970 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:24:24.970 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:24.970 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T18:24:24.970 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:24.970 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:24.970 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-03-25 16:40:19.623534637 +0000 2024-03-25T18:24:24.970 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-03-25 16:32:19.356759398 +0000 2024-03-25T18:24:24.970 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-03-25 16:32:19.356759398 +0000 2024-03-25T18:24:24.970 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-03-25 16:32:19.356759398 +0000 2024-03-25T18:24:24.970 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:24:25.036 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-03-25T18:24:25.036 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-03-25T18:24:25.036 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000215597 s, 2.4 MB/s 2024-03-25T18:24:25.037 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:24:25.093 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_3 2024-03-25T18:24:25.148 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:24:25.148 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:25.149 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T18:24:25.149 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:25.149 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:25.149 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-03-25 16:40:19.623534637 +0000 2024-03-25T18:24:25.149 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-03-25 16:32:19.691751111 +0000 2024-03-25T18:24:25.149 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-03-25 16:32:19.691751111 +0000 2024-03-25T18:24:25.149 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-03-25 16:32:19.691751111 +0000 2024-03-25T18:24:25.149 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:24:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-03-25T18:24:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-03-25T18:24:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000214821 s, 2.4 MB/s 2024-03-25T18:24:25.215 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:24:25.271 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_4 2024-03-25T18:24:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:24:25.325 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:25.325 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:24:25.326 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:25.326 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:25.326 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-03-25 16:40:19.623534637 +0000 2024-03-25T18:24:25.326 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-03-25 16:32:20.039742502 +0000 2024-03-25T18:24:25.326 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-03-25 16:32:20.039742502 +0000 2024-03-25T18:24:25.326 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-03-25 16:32:20.039742502 +0000 2024-03-25T18:24:25.326 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:24:25.390 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-03-25T18:24:25.390 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-03-25T18:24:25.391 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000173916 s, 2.9 MB/s 2024-03-25T18:24:25.392 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:24:25.447 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:25.447 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:24:25.447 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:24:25.466 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:24:25.466 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-03-25T18:24:25.523 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:24:25.523 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:25.523 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:24:25.524 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:25.524 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:25.524 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-25 16:38:40.918071457 +0000 2024-03-25T18:24:25.524 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-25 16:32:19.318078852 +0000 2024-03-25T18:24:25.524 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-25 16:32:19.318078852 +0000 2024-03-25T18:24:25.524 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-25 16:32:19.318078852 +0000 2024-03-25T18:24:25.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:24:25.592 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-25T18:24:25.593 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-25T18:24:25.593 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000269742 s, 1.9 MB/s 2024-03-25T18:24:25.594 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:24:25.651 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-03-25T18:24:25.706 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:24:25.707 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:25.707 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T18:24:25.707 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:25.707 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:25.707 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-25 16:38:40.918071457 +0000 2024-03-25T18:24:25.707 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-25 16:32:19.657068809 +0000 2024-03-25T18:24:25.707 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-25 16:32:19.657068809 +0000 2024-03-25T18:24:25.707 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-25 16:32:19.657068809 +0000 2024-03-25T18:24:25.707 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:24:25.773 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-25T18:24:25.774 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-25T18:24:25.774 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000201224 s, 2.5 MB/s 2024-03-25T18:24:25.775 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:24:25.831 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-03-25T18:24:25.887 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:24:25.887 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:25.887 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T18:24:25.887 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:25.887 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:25.887 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-25 16:38:40.919071428 +0000 2024-03-25T18:24:25.887 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-25 16:32:20.009058381 +0000 2024-03-25T18:24:25.887 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-25 16:32:20.009058381 +0000 2024-03-25T18:24:25.887 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-25 16:32:20.009058381 +0000 2024-03-25T18:24:25.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:24:25.953 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-25T18:24:25.953 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-25T18:24:25.953 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000227949 s, 2.2 MB/s 2024-03-25T18:24:25.954 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:24:26.010 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-03-25T18:24:26.066 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:24:26.066 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:24:26.066 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T18:24:26.066 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:24:26.066 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:24:26.066 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-25 16:38:40.919071428 +0000 2024-03-25T18:24:26.066 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-25 16:32:20.355048131 +0000 2024-03-25T18:24:26.066 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-25 16:32:20.355048131 +0000 2024-03-25T18:24:26.066 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-25 16:32:20.355048131 +0000 2024-03-25T18:24:26.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:24:26.131 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-25T18:24:26.131 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-25T18:24:26.131 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000206414 s, 2.5 MB/s 2024-03-25T18:24:26.132 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:24:26.188 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:26.188 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:24:26.188 INFO:tasks.ceph:Generating config... 2024-03-25T18:24:26.189 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:24:26.189 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:24:26.189 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:24:26.190 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:24:26.190 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:24:26.190 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:24:26.190 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:24:26.190 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:24:26.190 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:24:26.190 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:24:26.190 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:24:26.190 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:24:26.190 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:24:26.190 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:24:26.190 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:24:26.190 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:24:26.190 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:24:26.190 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:24:26.191 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:24:26.191 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:24:26.191 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:24:26.191 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:24:26.191 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:24:26.191 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:24:26.191 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:24:26.191 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:24:26.191 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:24:26.191 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:24:26.191 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:24:26.191 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:24:26.191 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:24:26.191 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:24:26.191 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:24:26.192 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:24:26.193 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:24:26.193 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:24:26.193 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:24:26.193 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:24:26.193 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:24:26.193 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:24:26.193 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:24:26.242 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:24:26.245 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:24:26.291 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:24:26.318 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.99:6789]'), ('mon.c', '[v1:172.21.15.99:6790]'), ('mon.b', '[v1:172.21.15.190:6789]')] 2024-03-25T18:24:26.318 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.99:6789],[v1:172.21.15.99:6790],[v1:172.21.15.190:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:24:26.319 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:26.319 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:24:26.373 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.99:6789]' --addv c '[v1:172.21.15.99:6790]' --addv b '[v1:172.21.15.190:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:26.458 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:26.458 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: generated fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:24:26.458 INFO:teuthology.orchestra.run.smithi099.stdout:setting min_mon_release = quincy 2024-03-25T18:24:26.458 INFO:teuthology.orchestra.run.smithi099.stdout:epoch 0 2024-03-25T18:24:26.458 INFO:teuthology.orchestra.run.smithi099.stdout:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:24:26.458 INFO:teuthology.orchestra.run.smithi099.stdout:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:24:26.458 INFO:teuthology.orchestra.run.smithi099.stdout:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:24:26.458 INFO:teuthology.orchestra.run.smithi099.stdout:min_mon_release 17 (quincy) 2024-03-25T18:24:26.458 INFO:teuthology.orchestra.run.smithi099.stdout:election_strategy: 3 2024-03-25T18:24:26.459 INFO:teuthology.orchestra.run.smithi099.stdout:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:24:26.459 INFO:teuthology.orchestra.run.smithi099.stdout:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:24:26.459 INFO:teuthology.orchestra.run.smithi099.stdout:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:24:26.459 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:24:26.460 DEBUG:teuthology.orchestra.run.smithi099:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:24:26.474 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 71c41715-c00e-41a8-a30d-ef4898065160... 2024-03-25T18:24:26.475 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:24:26.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:24:26.565 INFO:teuthology.orchestra.run.smithi099.stdout:[global] 2024-03-25T18:24:26.565 INFO:teuthology.orchestra.run.smithi099.stdout: chdir = "" 2024-03-25T18:24:26.565 INFO:teuthology.orchestra.run.smithi099.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:24:26.565 INFO:teuthology.orchestra.run.smithi099.stdout: auth supported = cephx 2024-03-25T18:24:26.565 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.565 INFO:teuthology.orchestra.run.smithi099.stdout: filestore xattr use omap = true 2024-03-25T18:24:26.565 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.565 INFO:teuthology.orchestra.run.smithi099.stdout: mon clock drift allowed = 1.000 2024-03-25T18:24:26.565 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.565 INFO:teuthology.orchestra.run.smithi099.stdout: osd crush chooseleaf type = 0 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: auth debug = true 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on old message = true 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on bug = true 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: mon pg warn max object skew = 0 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default size = 2 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow primary affinity = true 2024-03-25T18:24:26.566 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow pg remap = true 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on no sortbitwise = false 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on too few osds = false 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: mon_allow_pool_size_one = true 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: osd default data pool replay window = 5 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: mon allow pool delete = true 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.567 INFO:teuthology.orchestra.run.smithi099.stdout: mon cluster log file level = debug 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout: debug asserts on shutdown = true 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout: mon health detail to clog = false 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout: mon host = "[v1:172.21.15.99:6789],[v1:172.21.15.99:6790],[v1:172.21.15.190:6789]" 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout: mon client directed command retry = 5 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout: mon election default strategy = 3 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout: ms bind msgr2 = False 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout: ms inject socket failures = 5000 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout: ms type = async 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout: osd heartbeat grace = 80 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout: fsid = 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout:[osd] 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout: osd journal size = 100 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.568 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub load threshold = 5.0 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub max interval = 600 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: osd recover clone overlap = true 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: osd recovery max chunk = 1048576 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug shutdown = true 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug op order = true 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug verify stray on activate = true 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug trim objects = true 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: osd open classes on start = true 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug pg log writeout = true 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:24:26.569 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: osd map max advance = 10 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: journal zero on create = true 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: filestore apply finisher threads = 3 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: bdev debug aio = true 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug misdirected ops = true 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: bdev async discard = True 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: bdev enable discard = True 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: bluestore allocator = stupid 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: bluestore block size = 96636764160 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: bluestore fsck on mount = True 2024-03-25T18:24:26.570 INFO:teuthology.orchestra.run.smithi099.stdout: bluestore zero block detection = True 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: debug bluefs = 1 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: debug bluestore = 1 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: debug monc = 20 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: debug osd = 20 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: debug rocksdb = 10 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd full ratio = 0.9 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: osd class default list = * 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: osd class load list = * 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: osd client watch timeout = 120 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug verify cached snaps = True 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug verify missing on start = True 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:24:26.571 INFO:teuthology.orchestra.run.smithi099.stdout: osd fast shutdown = False 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: osd max markdown count = 1000 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: osd mclock override recovery settings = True 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: osd objectstore = bluestore 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: osd op queue = debug_random 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: osd op queue cut off = debug_random 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: osd shutdown pgref assert = True 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr] 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: debug mgr = 20 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: mgr/telemetry/nag = false 2024-03-25T18:24:26.572 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout:[mon] 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: debug paxos = 20 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: mon data avail warn = 5 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: mon mgr mkfs grace = 240 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd reporter subtree level = osd 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd prime pg temp = true 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:24:26.573 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: # 1m isn't quite enough 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: mon min osdmap epochs = 50 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd crush smoke test = False 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: mon osdmap full prune interval = 2 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: mon osdmap full prune min = 15 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: mon scrub interval = 300 2024-03-25T18:24:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on pool no app = False 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout: paxos service trim min = 10 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout:[client] 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout: rgw cache enabled = true 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable ops log = true 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable usage log = true 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout: debug monc = 20 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout: debug objecter = 20 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout: debug rados = 20 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.a] 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.c] 2024-03-25T18:24:26.575 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.b] 2024-03-25T18:24:26.576 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2024-03-25T18:24:26.576 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2024-03-25T18:24:26.576 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:24:26.576 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2024-03-25T18:24:26.576 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.576 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2024-03-25T18:24:26.576 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.576 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2024-03-25T18:24:26.576 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.576 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2024-03-25T18:24:26.576 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2024-03-25T18:24:26.576 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2024-03-25T18:24:26.577 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:24:26.578 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2024-03-25T18:24:26.578 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:24:26.578 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:24:26.578 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2024-03-25T18:24:26.578 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.578 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:24:26.578 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.578 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2024-03-25T18:24:26.578 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.578 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2024-03-25T18:24:26.578 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.578 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2024-03-25T18:24:26.578 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout: mon health detail to clog = false 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = "[v1:172.21.15.99:6789],[v1:172.21.15.99:6790],[v1:172.21.15.190:6789]" 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout: mon client directed command retry = 5 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout: mon election default strategy = 3 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout: ms bind msgr2 = False 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject socket failures = 5000 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout: ms type = async 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout: osd heartbeat grace = 80 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2024-03-25T18:24:26.579 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 600 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug trim objects = true 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:24:26.580 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: bdev async discard = True 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: bdev enable discard = True 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore allocator = stupid 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore block size = 96636764160 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore fsck on mount = True 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore zero block detection = True 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluefs = 1 2024-03-25T18:24:26.581 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluestore = 1 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: debug monc = 20 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 20 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: debug rocksdb = 10 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd full ratio = 0.9 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: osd class default list = * 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: osd class load list = * 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: osd client watch timeout = 120 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify cached snaps = True 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify missing on start = True 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: osd fast shutdown = False 2024-03-25T18:24:26.582 INFO:teuthology.orchestra.run.smithi190.stdout: osd max markdown count = 1000 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock override recovery settings = True 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout: osd objectstore = bluestore 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout: osd op queue = debug_random 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout: osd op queue cut off = debug_random 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout: osd shutdown pgref assert = True 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.583 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: # 1m isn't quite enough 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: mon min osdmap epochs = 50 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd crush smoke test = False 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune interval = 2 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune min = 15 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: mon scrub interval = 300 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on pool no app = False 2024-03-25T18:24:26.585 INFO:teuthology.orchestra.run.smithi190.stdout: paxos service trim min = 10 2024-03-25T18:24:26.586 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-25T18:24:26.586 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2024-03-25T18:24:26.586 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2024-03-25T18:24:26.586 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2024-03-25T18:24:26.586 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2024-03-25T18:24:26.586 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:24:26.586 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:24:26.586 INFO:teuthology.orchestra.run.smithi190.stdout: debug monc = 20 2024-03-25T18:24:26.586 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-03-25T18:24:26.586 INFO:teuthology.orchestra.run.smithi190.stdout: debug objecter = 20 2024-03-25T18:24:26.586 INFO:teuthology.orchestra.run.smithi190.stdout: debug rados = 20 2024-03-25T18:24:26.586 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2024-03-25T18:24:26.586 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.c] 2024-03-25T18:24:26.586 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.b] 2024-03-25T18:24:26.587 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:24:26.588 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:24:26.642 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:24:26.643 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:26.643 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:24:26.657 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:26.657 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:24:26.712 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi099.front.sepia.ceph.com 2024-03-25T18:24:26.712 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:26.712 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:24:26.712 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:24:26.791 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:26.792 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:26.846 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2024-03-25T18:24:26.846 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:24:26.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:24:26.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:24:26.886 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:24:26.886 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:26.940 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:24:26.940 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:24:26.941 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:24:26.995 INFO:teuthology.orchestra.run.smithi099.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:24:26.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:24:27.061 INFO:teuthology.orchestra.run.smithi190.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:24:27.064 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:24:27.064 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:24:27.064 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:24:27.106 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:24:27.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:24:27.163 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:24:27.178 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:24:27.179 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:24:27.179 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:24:27.205 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:27.205 INFO:tasks.ceph:role: osd.0 2024-03-25T18:24:27.205 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-25T18:24:27.205 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:24:27.279 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:27.279 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:27.279 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:27.280 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:27.280 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:27.280 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:27.280 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:27.795 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2024-03-25T18:24:27.797 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2024-03-25T18:24:27.797 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:24:27.948 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:24:27.983 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:24:28.051 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:28.051 INFO:tasks.ceph:role: osd.1 2024-03-25T18:24:28.051 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-25T18:24:28.051 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:24:28.125 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:28.125 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:28.125 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:28.126 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:28.126 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:28.126 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:28.126 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:28.126 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:28.126 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:28.126 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:28.574 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2024-03-25T18:24:28.576 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2024-03-25T18:24:28.576 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:24:28.669 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:24:28.702 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:24:28.770 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:28.770 INFO:tasks.ceph:role: osd.2 2024-03-25T18:24:28.770 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-25T18:24:28.770 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:24:28.844 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:28.844 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:28.844 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:28.844 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:28.844 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:28.844 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:28.844 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:28.845 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:28.845 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:28.845 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:29.337 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2024-03-25T18:24:29.338 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2024-03-25T18:24:29.339 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:24:29.455 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:24:29.489 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:24:29.556 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:29.556 INFO:tasks.ceph:role: osd.3 2024-03-25T18:24:29.556 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-25T18:24:29.556 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:24:29.631 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:29.631 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:29.631 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:29.632 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:29.632 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:29.632 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:29.632 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:30.150 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2024-03-25T18:24:30.152 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2024-03-25T18:24:30.152 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:24:30.270 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:24:30.304 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:30.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:30.399+0000 7f386080b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:24:30.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:30.399+0000 7f386080b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:24:30.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:30.399+0000 7f386080b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:24:30.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:30.399+0000 7f386080b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:24:30.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:30.399+0000 7f386080b640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:24:30.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:30.400+0000 7f386080b640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:24:33.251 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:24:33.282 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:33.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:33.371+0000 7f723ae68640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:24:33.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:33.371+0000 7f723ae68640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:24:33.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:33.371+0000 7f723ae68640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:24:33.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:33.371+0000 7f723ae68640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:24:33.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:33.371+0000 7f723ae68640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:24:33.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:33.372+0000 7f723ae68640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:24:36.219 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:24:36.251 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:36.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:36.341+0000 7f4abd611640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:24:36.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:36.341+0000 7f4abd611640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:24:36.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:36.341+0000 7f4abd611640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:24:36.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:36.341+0000 7f4abd611640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:24:36.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:36.341+0000 7f4abd611640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:24:36.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:36.342+0000 7f4abd611640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:24:39.199 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:24:39.229 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:39.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:39.320+0000 7f650928b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:24:39.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:39.320+0000 7f650928b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:24:39.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:39.320+0000 7f650928b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:24:39.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:39.320+0000 7f650928b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:24:39.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:39.320+0000 7f650928b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:24:39.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:39.320+0000 7f650928b640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:24:42.165 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:24:42.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:24:42.228 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:42.228 INFO:tasks.ceph:role: osd.4 2024-03-25T18:24:42.228 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-25T18:24:42.228 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:24:42.304 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:42.304 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:42.304 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:42.304 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:42.304 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:42.304 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:42.304 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:42.304 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:42.305 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:42.305 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:42.817 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-03-25T18:24:42.819 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-03-25T18:24:42.819 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:24:42.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:24:42.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:24:43.065 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:43.065 INFO:tasks.ceph:role: osd.5 2024-03-25T18:24:43.065 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-25T18:24:43.065 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:24:43.140 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:43.141 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:43.141 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:43.141 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:43.141 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:43.141 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:43.141 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:43.141 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:43.141 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:43.141 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:43.595 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-03-25T18:24:43.597 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-03-25T18:24:43.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:24:43.689 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:24:43.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:24:43.790 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:43.790 INFO:tasks.ceph:role: osd.6 2024-03-25T18:24:43.790 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-25T18:24:43.790 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:24:43.864 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:43.864 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:43.865 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:43.865 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:43.865 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:43.865 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:43.865 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:43.865 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:43.865 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:43.865 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:44.352 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-03-25T18:24:44.354 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-03-25T18:24:44.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:24:44.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:24:44.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:24:44.580 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:24:44.580 INFO:tasks.ceph:role: osd.7 2024-03-25T18:24:44.580 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-25T18:24:44.580 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:24:44.656 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:24:44.656 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:24:44.656 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:24:44.656 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:24:44.656 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:24:44.656 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:24:44.657 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:24:44.657 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:24:44.657 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:24:44.657 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:24:45.175 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-03-25T18:24:45.177 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-03-25T18:24:45.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:24:45.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:24:45.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:45.476+0000 7f3879a13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:24:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:45.477+0000 7f3879a13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:24:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:45.477+0000 7f3879a13640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:24:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:45.477+0000 7f3879a13640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:24:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:45.477+0000 7f3879a13640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:24:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:45.477+0000 7f3879a13640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:24:48.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:24:48.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:48.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:48.446+0000 7fafa1c9d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:24:48.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:48.447+0000 7fafa1c9d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:24:48.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:48.447+0000 7fafa1c9d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:24:48.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:48.447+0000 7fafa1c9d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:24:48.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:48.447+0000 7fafa1c9d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:24:48.448 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:48.447+0000 7fafa1c9d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:24:51.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:24:51.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:51.421 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:51.420+0000 7fd6ad822640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:24:51.421 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:51.420+0000 7fd6ad822640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:24:51.421 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:51.420+0000 7fd6ad822640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:24:51.421 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:51.420+0000 7fd6ad822640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:24:51.421 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:51.420+0000 7fd6ad822640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:24:51.422 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:51.420+0000 7fd6ad822640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:24:54.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:24:54.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:54.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:54.383+0000 7f89ce022640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:24:54.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:54.383+0000 7f89ce022640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:24:54.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:54.383+0000 7f89ce022640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:24:54.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:54.383+0000 7f89ce022640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:24:54.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:54.383+0000 7f89ce022640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:24:54.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:24:54.384+0000 7f89ce022640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:24:57.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:24:57.264 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:24:57.264 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:57.264 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:24:57.296 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:57.296 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:24:57.365 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:57.365 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:24:57.434 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:57.434 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:24:57.503 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:57.503 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:24:57.571 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:24:57.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:24:57.602 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:24:57.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:24:57.672 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:24:57.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:24:57.743 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:24:57.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:24:57.811 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:24:57.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:24:57.879 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:24:57.879 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:24:57.896 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:24:57.896 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:24:57.935 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:24:57.935 DEBUG:teuthology.orchestra.run.smithi099:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:24:57.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:24:57.977 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr.y] 2024-03-25T18:24:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBawQFmDxlSOxAAGL2veLF5z0JiQiGvRXkuoA== 2024-03-25T18:24:57.978 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.0] 2024-03-25T18:24:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBewQFmgEfeFxAAYuafaGtppxoKmfEsUsGOkw== 2024-03-25T18:24:57.978 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.1] 2024-03-25T18:24:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBhwQFmisQzFhAAlChh7ZJw0uu9hJuk+83mug== 2024-03-25T18:24:57.978 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.2] 2024-03-25T18:24:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBkwQFmywJrFBAAkX0JK4yAQV7OiNV/bDYy8Q== 2024-03-25T18:24:57.978 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.3] 2024-03-25T18:24:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBnwQFmd2klExAAJv3Dm9DED7adeRI/f6QhTg== 2024-03-25T18:24:57.978 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr.x] 2024-03-25T18:24:57.978 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBbwQFm5i+3AxAA9A6HHU566ih0NbpwzB/kwg== 2024-03-25T18:24:57.978 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.4] 2024-03-25T18:24:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBtwQFmeO98HBAAvSuYDRs35h1pd6tl3gNUKQ== 2024-03-25T18:24:57.979 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.5] 2024-03-25T18:24:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBwwQFm2fqyGhAAFR3CZY6XHlmt6M12OTzqrw== 2024-03-25T18:24:57.979 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.6] 2024-03-25T18:24:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBzwQFmHGEZGRAAQrhSQU8QW7JDpXW6jebukA== 2024-03-25T18:24:57.979 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.7] 2024-03-25T18:24:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQB2wQFmjLnnFhAAcOdVEF/u3Z8sdpZ2Q4sH/w== 2024-03-25T18:24:57.979 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2024-03-25T18:24:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBbwQFm671jBhAAyQOKycDbnOUrD7v8ORrnzQ== 2024-03-25T18:24:57.979 INFO:teuthology.orchestra.run.smithi099.stdout:[client.1] 2024-03-25T18:24:57.979 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBbwQFmPB7HCRAAhyAFSZuOsIO1owukEogSbA== 2024-03-25T18:24:58.000 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.y] 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBawQFmDxlSOxAAGL2veLF5z0JiQiGvRXkuoA== 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.0] 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBewQFmgEfeFxAAYuafaGtppxoKmfEsUsGOkw== 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.1] 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBhwQFmisQzFhAAlChh7ZJw0uu9hJuk+83mug== 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.2] 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBkwQFmywJrFBAAkX0JK4yAQV7OiNV/bDYy8Q== 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.3] 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBnwQFmd2klExAAJv3Dm9DED7adeRI/f6QhTg== 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.x] 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBbwQFm5i+3AxAA9A6HHU566ih0NbpwzB/kwg== 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.4] 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBtwQFmeO98HBAAvSuYDRs35h1pd6tl3gNUKQ== 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.5] 2024-03-25T18:24:58.001 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBwwQFm2fqyGhAAFR3CZY6XHlmt6M12OTzqrw== 2024-03-25T18:24:58.002 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.6] 2024-03-25T18:24:58.002 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBzwQFmHGEZGRAAQrhSQU8QW7JDpXW6jebukA== 2024-03-25T18:24:58.002 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.7] 2024-03-25T18:24:58.002 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQB2wQFmjLnnFhAAcOdVEF/u3Z8sdpZ2Q4sH/w== 2024-03-25T18:24:58.002 INFO:teuthology.orchestra.run.smithi190.stdout:[client.0] 2024-03-25T18:24:58.002 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBbwQFm671jBhAAyQOKycDbnOUrD7v8ORrnzQ== 2024-03-25T18:24:58.002 INFO:teuthology.orchestra.run.smithi190.stdout:[client.1] 2024-03-25T18:24:58.002 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBbwQFmPB7HCRAAhyAFSZuOsIO1owukEogSbA== 2024-03-25T18:24:58.003 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:24:58.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:24:58.095 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.151 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.205 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.259 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.312 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:24:58.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:24:58.365 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.419 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.472 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.474 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.526 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:24:58.579 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:24:58.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:24:58.636 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:24:58.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:24:58.692 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:24:58.692 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:24:58.718 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:24:58.817 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:24:58.844 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:24:58.911 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:24:59.008 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:24:59.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:24:59.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:24:59.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:24:59.207 DEBUG:teuthology.orchestra.run.smithi099:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:59.210 DEBUG:teuthology.orchestra.run.smithi190:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:24:59.262 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:24:59.262 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:24:59.262 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:24:59.263 DEBUG:teuthology.orchestra.run.smithi099:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:24:59.265 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:24:59.265 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:24:59.265 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:24:59.266 DEBUG:teuthology.orchestra.run.smithi099:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:24:59.268 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:24:59.268 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:24:59.268 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:24:59.268 DEBUG:teuthology.orchestra.run.smithi190:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:24:59.305 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:24:59.305 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:24:59.305 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:24:59.306 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:24:59.308 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:24:59.308 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:24:59.308 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:24:59.310 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:24:59.310 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:24:59.310 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:24:59.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.410+0000 7ffb362d9640 10 monclient: get_monmap_and_config 2024-03-25T18:24:59.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.410+0000 7ffb362d9640 10 monclient: build_initial_monmap 2024-03-25T18:24:59.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.410+0000 7ffb362d9640 10 monclient: monmap: 2024-03-25T18:24:59.411 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 0 2024-03-25T18:24:59.411 INFO:teuthology.orchestra.run.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:24:59.411 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2024-03-25T18:24:59.411302+0000 2024-03-25T18:24:59.411 INFO:teuthology.orchestra.run.smithi099.stderr:created 2024-03-25T18:24:59.411302+0000 2024-03-25T18:24:59.411 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:59.412 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2024-03-25T18:24:59.412 INFO:teuthology.orchestra.run.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:24:59.412 INFO:teuthology.orchestra.run.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:24:59.412 INFO:teuthology.orchestra.run.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:24:59.412 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:24:59.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.410+0000 7ffb362d9640 1 Processor -- start 2024-03-25T18:24:59.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb362d9640 1 -- start start 2024-03-25T18:24:59.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb362d9640 10 monclient: init 2024-03-25T18:24:59.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb362d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:59.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb362d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:59.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb362d9640 10 monclient(hunting): picked mon.noname-a con 0x7ffb300b4700 addr v1:172.21.15.99:6789/0 2024-03-25T18:24:59.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb362d9640 10 monclient(hunting): picked mon.noname-c con 0x7ffb30142a20 addr v1:172.21.15.190:6789/0 2024-03-25T18:24:59.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb362d9640 10 monclient(hunting): picked mon.noname-b con 0x7ffb300a96b0 addr v1:172.21.15.99:6790/0 2024-03-25T18:24:59.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb362d9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300a9d60 con 0x7ffb300b4700 2024-03-25T18:24:59.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb362d9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb3013ff10 con 0x7ffb30142a20 2024-03-25T18:24:59.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb362d9640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300aa1f0 con 0x7ffb300a96b0 2024-03-25T18:24:59.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb362d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:59.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb362d9640 10 monclient(hunting): authenticate will time out at 7317.579506s 2024-03-25T18:24:59.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb2ffff640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb300b4700 legacy=0x7ffb30142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2024-03-25T18:24:59.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb34876640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb3013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6790/0 2024-03-25T18:24:59.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.411+0000 7ffb2f7fe640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb3013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-03-25T18:24:59.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.612+0000 7ffb2ffff640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb300b4700 legacy=0x7ffb30142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2024-03-25T18:24:59.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.612+0000 7ffb34876640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb3013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6790/0 2024-03-25T18:24:59.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:24:59.612+0000 7ffb2f7fe640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb3013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-03-25T18:25:00.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:00.013+0000 7ffb2ffff640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb300b4700 legacy=0x7ffb30142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2024-03-25T18:25:00.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:00.013+0000 7ffb34876640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb3013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6790/0 2024-03-25T18:25:00.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:00.013+0000 7ffb2f7fe640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb3013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-03-25T18:25:00.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:00.814+0000 7ffb2ffff640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb300b4700 legacy=0x7ffb30142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2024-03-25T18:25:00.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:00.814+0000 7ffb34876640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb3013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6790/0 2024-03-25T18:25:00.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:00.814+0000 7ffb2f7fe640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb3013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-03-25T18:25:02.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 10 monclient(hunting): tick 2024-03-25T18:25:02.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:25:02.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:25:02.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:25:02.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb3013f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:02.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb3013f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:02.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb300b4700 legacy=0x7ffb30142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:02.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:02.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-a con 0x7ffb1c003680 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:02.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-c con 0x7ffb30142a20 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:02.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-b con 0x7ffb300a96b0 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:02.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300b4ae0 con 0x7ffb1c003680 2024-03-25T18:25:02.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2ffff640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb1c003680 legacy=0x7ffb1c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2024-03-25T18:25:02.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300a9d60 con 0x7ffb30142a20 2024-03-25T18:25:02.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb3013ff10 con 0x7ffb300a96b0 2024-03-25T18:25:02.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2dffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:02.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb34876640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb1c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6790/0 2024-03-25T18:25:02.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.411+0000 7ffb2f7fe640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb1c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-03-25T18:25:02.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.612+0000 7ffb2ffff640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb1c003680 legacy=0x7ffb1c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2024-03-25T18:25:02.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.612+0000 7ffb34876640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb1c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6790/0 2024-03-25T18:25:02.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:02.612+0000 7ffb2f7fe640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb1c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-03-25T18:25:03.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:03.013+0000 7ffb34876640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb1c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6790/0 2024-03-25T18:25:03.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:03.013+0000 7ffb2ffff640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb1c003680 legacy=0x7ffb1c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2024-03-25T18:25:03.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:03.013+0000 7ffb2f7fe640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb1c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-03-25T18:25:03.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:03.814+0000 7ffb2ffff640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb1c003680 legacy=0x7ffb1c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2024-03-25T18:25:03.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:03.814+0000 7ffb34876640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb1c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6790/0 2024-03-25T18:25:03.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:03.814+0000 7ffb2f7fe640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb1c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 10 monclient(hunting): tick 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb1c00cbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb1c009300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:05.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb1c003680 legacy=0x7ffb1c005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:05.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:05.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-b con 0x7ffb300a96b0 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:05.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-c con 0x7ffb30142a20 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:05.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-a con 0x7ffb1c003680 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:05.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300aa1f0 con 0x7ffb1c003680 2024-03-25T18:25:05.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300b4ae0 con 0x7ffb30142a20 2024-03-25T18:25:05.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300a9d60 con 0x7ffb300a96b0 2024-03-25T18:25:05.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.411+0000 7ffb2dffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:05.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.412+0000 7ffb2ffff640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb1c010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6790/0 2024-03-25T18:25:05.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.412+0000 7ffb34876640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb1c003680 legacy=0x7ffb1c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2024-03-25T18:25:05.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.412+0000 7ffb2f7fe640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb1c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-03-25T18:25:05.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.613+0000 7ffb2ffff640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb1c010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6790/0 2024-03-25T18:25:05.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.613+0000 7ffb34876640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb1c003680 legacy=0x7ffb1c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2024-03-25T18:25:05.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:05.613+0000 7ffb2f7fe640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb1c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-03-25T18:25:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:06.014+0000 7ffb2ffff640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb1c010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6790/0 2024-03-25T18:25:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:06.014+0000 7ffb34876640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb1c003680 legacy=0x7ffb1c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2024-03-25T18:25:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:06.014+0000 7ffb2f7fe640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb1c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-03-25T18:25:06.320 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:00:06.878 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:06.616 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:00:07.209 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:06.619 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:00:07.212 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:06.625 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:00:07.184 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:06.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:06.815+0000 7ffb2ffff640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb1c010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6790/0 2024-03-25T18:25:06.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:06.815+0000 7ffb34876640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb1c003680 legacy=0x7ffb1c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2024-03-25T18:25:06.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:06.815+0000 7ffb2f7fe640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb1c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.190:6789/0 2024-03-25T18:25:06.930 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:00:07.522 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:06.932 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:00:07.525 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:08.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 10 monclient(hunting): tick 2024-03-25T18:25:08.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:25:08.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:25:08.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:25:08.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 1 -- >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb1c010430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:08.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 1 -- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb1c00cbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:08.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 1 -- >> v1:172.21.15.99:6789/0 conn(0x7ffb1c003680 legacy=0x7ffb1c005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:08.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:08.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-b con 0x7ffb300a96b0 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:08.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-a con 0x7ffb1c003680 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:08.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-c con 0x7ffb30142a20 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:08.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb3013ff10 con 0x7ffb1c003680 2024-03-25T18:25:08.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300aa1f0 con 0x7ffb30142a20 2024-03-25T18:25:08.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300b4ae0 con 0x7ffb300a96b0 2024-03-25T18:25:08.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.412+0000 7ffb2dffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:08.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.500+0000 7ffb34876640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 0x7ffb1c00cbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:37868/0 (socket says 172.21.15.99:37868) 2024-03-25T18:25:08.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:08.500+0000 7ffb34876640 1 -- 172.21.15.99:0/3079406589 learned_addr learned my addr 172.21.15.99:0/3079406589 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:25:11.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 10 monclient(hunting): tick 2024-03-25T18:25:11.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:25:11.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:25:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:25:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 1 -- 172.21.15.99:0/3079406589 >> v1:172.21.15.99:6790/0 conn(0x7ffb300a96b0 legacy=0x7ffb1c0092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 1 -- 172.21.15.99:0/3079406589 >> v1:172.21.15.190:6789/0 conn(0x7ffb30142a20 legacy=0x7ffb1c00cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 1 -- 172.21.15.99:0/3079406589 >> v1:172.21.15.99:6789/0 conn(0x7ffb1c003680 legacy=0x7ffb1c010470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-b con 0x7ffb1c005ba0 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-c con 0x7ffb1c012500 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-a con 0x7ffb1c007270 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 1 -- 172.21.15.99:0/3079406589 --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300a9d60 con 0x7ffb1c007270 2024-03-25T18:25:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 1 -- 172.21.15.99:0/3079406589 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c0093c0 con 0x7ffb1c012500 2024-03-25T18:25:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 1 -- 172.21.15.99:0/3079406589 --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c0095f0 con 0x7ffb1c005ba0 2024-03-25T18:25:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:11.412+0000 7ffb2dffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 10 monclient(hunting): tick 2024-03-25T18:25:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:25:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:25:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:25:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 1 -- 172.21.15.99:0/3079406589 >> v1:172.21.15.99:6790/0 conn(0x7ffb1c005ba0 legacy=0x7ffb1c010c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 1 -- 172.21.15.99:0/3079406589 >> v1:172.21.15.190:6789/0 conn(0x7ffb1c012500 legacy=0x7ffb1c0148f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 1 -- 172.21.15.99:0/3079406589 >> v1:172.21.15.99:6789/0 conn(0x7ffb1c007270 legacy=0x7ffb1c010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-b con 0x7ffb1c0062a0 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-c con 0x7ffb1c012500 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 10 monclient(hunting): picked mon.noname-a con 0x7ffb1c006e30 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:14.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 1 -- 172.21.15.99:0/3079406589 --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c009820 con 0x7ffb1c006e30 2024-03-25T18:25:14.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 1 -- 172.21.15.99:0/3079406589 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c0104d0 con 0x7ffb1c012500 2024-03-25T18:25:14.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 1 -- 172.21.15.99:0/3079406589 --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c0106b0 con 0x7ffb1c0062a0 2024-03-25T18:25:14.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:14.412+0000 7ffb2dffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.894+0000 7ffb2effd640 1 -- 172.21.15.99:0/3079406589 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294246733 0 0) 0x7ffb1c0106b0 con 0x7ffb1c0062a0 2024-03-25T18:25:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.894+0000 7ffb2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.894+0000 7ffb2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.894+0000 7ffb2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.894+0000 7ffb2effd640 10 monclient(hunting): my global_id is 4103 2024-03-25T18:25:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.894+0000 7ffb2effd640 1 -- 172.21.15.99:0/3079406589 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb18003520 con 0x7ffb1c0062a0 2024-03-25T18:25:16.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.900+0000 7ffb2effd640 1 -- 172.21.15.99:0/3079406589 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962103597 0 0) 0x7ffb1c009820 con 0x7ffb1c006e30 2024-03-25T18:25:16.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.900+0000 7ffb2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.900+0000 7ffb2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.900+0000 7ffb2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.900+0000 7ffb2effd640 10 monclient(hunting): my global_id is 4104 2024-03-25T18:25:16.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.900+0000 7ffb2effd640 1 -- 172.21.15.99:0/3079406589 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1c0106b0 con 0x7ffb1c006e30 2024-03-25T18:25:16.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.919+0000 7ffb2effd640 1 -- 172.21.15.99:0/3079406589 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970702066 0 0) 0x7ffb1c0104d0 con 0x7ffb1c012500 2024-03-25T18:25:16.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.919+0000 7ffb2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.919+0000 7ffb2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.919+0000 7ffb2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.919+0000 7ffb2effd640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:25:16.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.919+0000 7ffb2effd640 1 -- 172.21.15.99:0/3079406589 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1c009820 con 0x7ffb1c012500 2024-03-25T18:25:16.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 1 -- 172.21.15.99:0/3079406589 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1089804467 0 0) 0x7ffb1c0106b0 con 0x7ffb1c006e30 2024-03-25T18:25:16.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 1 -- 172.21.15.99:0/3079406589 >> v1:172.21.15.99:6790/0 conn(0x7ffb1c0062a0 legacy=0x7ffb1c006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 1 -- 172.21.15.99:0/3079406589 >> v1:172.21.15.190:6789/0 conn(0x7ffb1c012500 legacy=0x7ffb1c010c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 1 monclient: found mon.noname-a 2024-03-25T18:25:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:25:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 1 -- 172.21.15.99:0/3079406589 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c0108e0 con 0x7ffb1c006e30 2024-03-25T18:25:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 10 monclient: _finish_auth 0 2024-03-25T18:25:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 1 -- 172.21.15.99:0/3079406589 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ffb280042b0 con 0x7ffb1c006e30 2024-03-25T18:25:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb2effd640 10 monclient: dump: 2024-03-25T18:25:16.947 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2024-03-25T18:25:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:16.948 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:25:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb362d9640 5 monclient: authenticate success, global_id 4104 2024-03-25T18:25:16.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.945+0000 7ffb362d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:16.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.992+0000 7ffb2effd640 1 -- 172.21.15.99:0/3079406589 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb28004410 con 0x7ffb1c006e30 2024-03-25T18:25:16.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.992+0000 7ffb2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:16.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.992+0000 7ffb2effd640 1 -- 172.21.15.99:0/3079406589 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ffb28004860 con 0x7ffb1c006e30 2024-03-25T18:25:16.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.992+0000 7ffb2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.992+0000 7ffb2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:16.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.992+0000 7ffb2effd640 10 monclient: dump: 2024-03-25T18:25:16.994 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2024-03-25T18:25:16.994 INFO:teuthology.orchestra.run.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:16.994 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:16.994 INFO:teuthology.orchestra.run.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:16.994 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.994 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:16.994 INFO:teuthology.orchestra.run.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:16.994 INFO:teuthology.orchestra.run.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:16.994 INFO:teuthology.orchestra.run.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:16.994 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:25:16.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.992+0000 7ffb362d9640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:16.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.992+0000 7ffb362d9640 4 set_mon_vals no callback set 2024-03-25T18:25:16.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.993+0000 7ffb362d9640 10 monclient: shutdown 2024-03-25T18:25:16.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.993+0000 7ffb362d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:16.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.993+0000 7ffb362d9640 1 -- 172.21.15.99:0/3079406589 >> v1:172.21.15.99:6789/0 conn(0x7ffb1c006e30 legacy=0x7ffb1c00cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.993+0000 7ffb362d9640 1 -- 172.21.15.99:0/3079406589 shutdown_connections 2024-03-25T18:25:16.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.993+0000 7ffb362d9640 1 -- 172.21.15.99:0/3079406589 >> 172.21.15.99:0/3079406589 conn(0x7ffb300668e0 msgr2=0x7ffb30066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:16.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.993+0000 7ffb362d9640 1 -- 172.21.15.99:0/3079406589 shutdown_connections 2024-03-25T18:25:16.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.993+0000 7ffb362d9640 1 -- 172.21.15.99:0/3079406589 wait complete. 2024-03-25T18:25:16.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.994+0000 7ffb362d9640 10 monclient: build_initial_monmap 2024-03-25T18:25:16.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.994+0000 7ffb362d9640 10 monclient: monmap: 2024-03-25T18:25:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 0 2024-03-25T18:25:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 0.000000 2024-03-25T18:25:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:created 0.000000 2024-03-25T18:25:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 1 2024-03-25T18:25:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:16.998 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:25:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.994+0000 7ffb362d9640 1 librados: starting msgr at 2024-03-25T18:25:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.994+0000 7ffb362d9640 1 librados: starting objecter 2024-03-25T18:25:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.994+0000 7ffb362d9640 1 Processor -- start 2024-03-25T18:25:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.994+0000 7ffb362d9640 1 -- start start 2024-03-25T18:25:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.994+0000 7ffb362d9640 1 librados: setting wanted keys 2024-03-25T18:25:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.994+0000 7ffb362d9640 1 librados: calling monclient init 2024-03-25T18:25:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.994+0000 7ffb362d9640 10 monclient: init 2024-03-25T18:25:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.995+0000 7ffb362d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.995+0000 7ffb362d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.995+0000 7ffb362d9640 10 monclient(hunting): picked mon.noname-b con 0x7ffb1c0062a0 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.995+0000 7ffb362d9640 10 monclient(hunting): picked mon.noname-c con 0x7ffb1c006e30 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.995+0000 7ffb362d9640 10 monclient(hunting): picked mon.noname-a con 0x7ffb1c012500 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.995+0000 7ffb362d9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb301d7c00 con 0x7ffb1c012500 2024-03-25T18:25:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.995+0000 7ffb362d9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb301d87a0 con 0x7ffb1c0062a0 2024-03-25T18:25:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.995+0000 7ffb362d9640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300b0840 con 0x7ffb1c006e30 2024-03-25T18:25:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.995+0000 7ffb362d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.995+0000 7ffb362d9640 10 monclient(hunting): authenticate will time out at 7335.163243s 2024-03-25T18:25:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.996+0000 7ffb34876640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7ffb1c012500 0x7ffb301e22d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:48572/0 (socket says 172.21.15.99:48572) 2024-03-25T18:25:17.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.996+0000 7ffb34876640 1 -- 172.21.15.99:0/2732412574 learned_addr learned my addr 172.21.15.99:0/2732412574 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:25:17.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.996+0000 7ffb2cff9640 10 client.?.objecter ms_handle_connect 0x7ffb1c012500 2024-03-25T18:25:17.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.996+0000 7ffb2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:17.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.996+0000 7ffb2cff9640 10 client.?.objecter ms_handle_connect 0x7ffb1c0062a0 2024-03-25T18:25:17.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:16.996+0000 7ffb2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:17.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.037+0000 7ffb2cff9640 10 client.?.objecter ms_handle_connect 0x7ffb1c006e30 2024-03-25T18:25:17.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.037+0000 7ffb2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:17.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.051+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684038751 0 0) 0x7ffb300b0840 con 0x7ffb1c006e30 2024-03-25T18:25:17.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.051+0000 7ffb2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:17.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.051+0000 7ffb2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:17.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.051+0000 7ffb2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:17.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.051+0000 7ffb2cff9640 10 monclient(hunting): my global_id is 4109 2024-03-25T18:25:17.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.051+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1c0106b0 con 0x7ffb1c006e30 2024-03-25T18:25:17.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.053+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457658797 0 0) 0x7ffb301d87a0 con 0x7ffb1c0062a0 2024-03-25T18:25:17.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.053+0000 7ffb2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:17.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.053+0000 7ffb2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:17.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.053+0000 7ffb2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:17.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.053+0000 7ffb2cff9640 10 monclient(hunting): my global_id is 4108 2024-03-25T18:25:17.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.053+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb300b0840 con 0x7ffb1c0062a0 2024-03-25T18:25:17.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.070+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953872275 0 0) 0x7ffb301d7c00 con 0x7ffb1c012500 2024-03-25T18:25:17.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.070+0000 7ffb2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:17.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.070+0000 7ffb2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:17.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.070+0000 7ffb2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:17.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.070+0000 7ffb2cff9640 10 monclient(hunting): my global_id is 4107 2024-03-25T18:25:17.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.070+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb301d87a0 con 0x7ffb1c012500 2024-03-25T18:25:17.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.095+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2480049381 0 0) 0x7ffb1c0106b0 con 0x7ffb1c006e30 2024-03-25T18:25:17.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.095+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb301d7c00 con 0x7ffb1c006e30 2024-03-25T18:25:17.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.095+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ffb240042e0 con 0x7ffb1c006e30 2024-03-25T18:25:17.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.095+0000 7ffb2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.095+0000 7ffb2cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.095+0000 7ffb2cff9640 10 monclient(hunting): dump: 2024-03-25T18:25:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2024-03-25T18:25:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:17.097 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:25:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.095+0000 7ffb2cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:17.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.196+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 773745859 0 0) 0x7ffb301d7c00 con 0x7ffb1c006e30 2024-03-25T18:25:17.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.196+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 >> v1:172.21.15.190:6789/0 conn(0x7ffb1c0062a0 legacy=0x7ffb301d8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:17.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.196+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 >> v1:172.21.15.99:6789/0 conn(0x7ffb1c012500 legacy=0x7ffb301e22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:17.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.196+0000 7ffb2cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:25:17.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.196+0000 7ffb2cff9640 1 monclient: found mon.c 2024-03-25T18:25:17.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.196+0000 7ffb2cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:17.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.196+0000 7ffb2cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:17.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.196+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb301d7890 con 0x7ffb1c006e30 2024-03-25T18:25:17.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.196+0000 7ffb2cff9640 10 monclient: _finish_auth 0 2024-03-25T18:25:17.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.196+0000 7ffb2cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:25:17.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.196+0000 7ffb2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:17.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.196+0000 7ffb362d9640 5 monclient: authenticate success, global_id 4109 2024-03-25T18:25:17.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.196+0000 7ffb362d9640 10 monclient: _renew_subs 2024-03-25T18:25:17.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.197+0000 7ffb362d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:17.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.197+0000 7ffb362d9640 1 -- 172.21.15.99:0/2732412574 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb300b1670 con 0x7ffb1c006e30 2024-03-25T18:25:17.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.197+0000 7ffb362d9640 10 client.4109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:17.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.197+0000 7ffb362d9640 10 monclient: _renew_subs 2024-03-25T18:25:17.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.197+0000 7ffb362d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:17.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.197+0000 7ffb362d9640 1 -- 172.21.15.99:0/2732412574 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb300ae170 con 0x7ffb1c006e30 2024-03-25T18:25:17.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.197+0000 7ffb362d9640 1 librados: init done 2024-03-25T18:25:17.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.197+0000 7ffb362d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:17.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.198+0000 7ffb362d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:17.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.198+0000 7ffb362d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:17.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.198+0000 7ffb362d9640 1 -- 172.21.15.99:0/2732412574 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb300b7540 con 0x7ffb1c006e30 2024-03-25T18:25:17.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb24004440 con 0x7ffb1c006e30 2024-03-25T18:25:17.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:17.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ffb24004850 con 0x7ffb1c006e30 2024-03-25T18:25:17.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:17.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:17.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 10 monclient: dump: 2024-03-25T18:25:17.399 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2024-03-25T18:25:17.399 INFO:teuthology.orchestra.run.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:17.399 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:17.400 INFO:teuthology.orchestra.run.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:17.400 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:17.400 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:17.400 INFO:teuthology.orchestra.run.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:17.400 INFO:teuthology.orchestra.run.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:17.400 INFO:teuthology.orchestra.run.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:17.400 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:25:17.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ffb24004e00 con 0x7ffb1c006e30 2024-03-25T18:25:17.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2e7fc640 4 set_mon_vals no callback set 2024-03-25T18:25:17.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 893207080 0 0) 0x7ffb24006630 con 0x7ffb1c006e30 2024-03-25T18:25:17.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 10 client.4109.objecter ms_dispatch 0x7ffb30000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:25:17.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 3 client.4109.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:25:17.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 3 client.4109.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:25:17.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 20 client.4109.objecter dump_active .. 0 homeless 2024-03-25T18:25:17.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ffb24006a70 con 0x7ffb1c006e30 2024-03-25T18:25:17.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:17.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.397+0000 7ffb2cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:17.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.464+0000 7ffb362d9640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:25:17.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.464+0000 7ffb362d9640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:25:17.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.464+0000 7ffb362d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:17.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.464+0000 7ffb362d9640 1 -- 172.21.15.99:0/2732412574 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ffb3004f730 con 0x7ffb1c006e30 2024-03-25T18:25:17.584 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:17.583+0000 7f761b873040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:25:17.639 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:17.638+0000 7f761b873040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:25:17.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.661+0000 7ffb2cff9640 1 -- 172.21.15.99:0/2732412574 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7ffb24005190 con 0x7ffb1c006e30 2024-03-25T18:25:17.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.661+0000 7ffb2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:25:17.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.661+0000 7ffb2cff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:25:17.662 INFO:teuthology.orchestra.run.smithi099.stderr:adjusted tunables profile to default 2024-03-25T18:25:17.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.662+0000 7ffb362d9640 10 librados: watch_flush enter 2024-03-25T18:25:17.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.662+0000 7ffb362d9640 10 librados: watch_flush exit 2024-03-25T18:25:17.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.662+0000 7ffb362d9640 20 client.4109.objecter shutdown clearing up homeless session... 2024-03-25T18:25:17.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.662+0000 7ffb362d9640 10 client.4109.objecter successfully canceled tick 2024-03-25T18:25:17.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.662+0000 7ffb362d9640 10 monclient: shutdown 2024-03-25T18:25:17.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.662+0000 7ffb362d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:17.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.662+0000 7ffb362d9640 1 -- 172.21.15.99:0/2732412574 >> v1:172.21.15.99:6790/0 conn(0x7ffb1c006e30 legacy=0x7ffb300b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:17.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.662+0000 7ffb362d9640 1 -- 172.21.15.99:0/2732412574 shutdown_connections 2024-03-25T18:25:17.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.662+0000 7ffb362d9640 1 -- 172.21.15.99:0/2732412574 >> 172.21.15.99:0/2732412574 conn(0x7ffb300668e0 msgr2=0x7ffb300b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:17.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.662+0000 7ffb362d9640 1 -- 172.21.15.99:0/2732412574 shutdown_connections 2024-03-25T18:25:17.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.662+0000 7ffb362d9640 1 -- 172.21.15.99:0/2732412574 wait complete. 2024-03-25T18:25:17.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:17.662+0000 7ffb362d9640 1 librados: shutdown 2024-03-25T18:25:17.678 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:25:17.678 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:25:17.678 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:25:17.678 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:25:17.708 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:25:17.708 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:25:17.762 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:17.761+0000 7ff11184b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:25:17.775 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:25:17.775 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:25:17.823 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:17.822+0000 7ff11184b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:25:17.842 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-25T18:25:17.843 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:25:17.910 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:25:17.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:25:17.939 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:25:17.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:25:18.006 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:25:18.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:25:18.073 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-25T18:25:18.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:25:18.140 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new b62a6f41-2c55-459e-b9f1-b90fc256eaea 0 2024-03-25T18:25:18.195 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:18.194+0000 7f761b873040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:25:18.254 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:18.253+0000 7f761b873040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:25:18.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.283+0000 7f289c23e640 10 monclient: get_monmap_and_config 2024-03-25T18:25:18.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.283+0000 7f289c23e640 10 monclient: build_initial_monmap 2024-03-25T18:25:18.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.283+0000 7f289c23e640 10 monclient: monmap: 2024-03-25T18:25:18.284 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:18.284 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:18.285 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:25:18.284611+0000 2024-03-25T18:25:18.285 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:25:18.284611+0000 2024-03-25T18:25:18.285 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:18.285 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:18.285 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:18.285 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:25:18.285 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:25:18.285 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.284+0000 7f289c23e640 1 Processor -- start 2024-03-25T18:25:18.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.284+0000 7f289c23e640 1 -- start start 2024-03-25T18:25:18.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.284+0000 7f289c23e640 10 monclient: init 2024-03-25T18:25:18.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.284+0000 7f289c23e640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:18.287 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.284+0000 7f289c23e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:18.287 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.284+0000 7f289c23e640 10 monclient(hunting): picked mon.noname-c con 0x7f28940b4700 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:18.287 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.284+0000 7f289c23e640 10 monclient(hunting): picked mon.noname-a con 0x7f2894142a20 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:18.287 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.284+0000 7f289c23e640 10 monclient(hunting): picked mon.noname-b con 0x7f28940a96b0 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:18.287 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.284+0000 7f289c23e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28940a9d60 con 0x7f2894142a20 2024-03-25T18:25:18.287 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.284+0000 7f289c23e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289413ff10 con 0x7f28940b4700 2024-03-25T18:25:18.287 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.284+0000 7f289c23e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28940aa1f0 con 0x7f28940a96b0 2024-03-25T18:25:18.287 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.284+0000 7f289c23e640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:18.287 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.284+0000 7f289c23e640 10 monclient(hunting): authenticate will time out at 7473.629619s 2024-03-25T18:25:18.287 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.285+0000 7f28997d9640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2894142a20 0x7f289413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.190:41334/0 (socket says 172.21.15.190:41334) 2024-03-25T18:25:18.287 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.285+0000 7f28997d9640 1 -- 172.21.15.190:0/3865403473 learned_addr learned my addr 172.21.15.190:0/3865403473 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:18.288 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.287+0000 7f2898fd8640 1 -- 172.21.15.190:0/3865403473 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037156345 0 0) 0x7f289413ff10 con 0x7f28940b4700 2024-03-25T18:25:18.288 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.287+0000 7f2898fd8640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.288 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.287+0000 7f2898fd8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.289 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.287+0000 7f2898fd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.289 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.287+0000 7f2898fd8640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:25:18.289 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.287+0000 7f2898fd8640 1 -- 172.21.15.190:0/3865403473 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2880003520 con 0x7f28940b4700 2024-03-25T18:25:18.289 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.287+0000 7f2898fd8640 1 -- 172.21.15.190:0/3865403473 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384481079 0 0) 0x7f28940a9d60 con 0x7f2894142a20 2024-03-25T18:25:18.289 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.287+0000 7f2898fd8640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.289 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.287+0000 7f2898fd8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.289 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.287+0000 7f2898fd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.289 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.287+0000 7f2898fd8640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:25:18.289 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.287+0000 7f2898fd8640 1 -- 172.21.15.190:0/3865403473 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f289413ff10 con 0x7f2894142a20 2024-03-25T18:25:18.292 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 1 -- 172.21.15.190:0/3865403473 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 153209228 0 0) 0x7f2880003520 con 0x7f28940b4700 2024-03-25T18:25:18.292 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 1 -- 172.21.15.190:0/3865403473 >> v1:172.21.15.99:6790/0 conn(0x7f28940a96b0 legacy=0x7f289413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:25:18.292 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 1 -- 172.21.15.190:0/3865403473 >> v1:172.21.15.99:6789/0 conn(0x7f2894142a20 legacy=0x7f289413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.292 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 10 monclient: _finish_hunting 0 2024-03-25T18:25:18.292 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 1 monclient: found mon.noname-c 2024-03-25T18:25:18.292 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:18.292 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:25:18.292 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 1 -- 172.21.15.190:0/3865403473 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28940b4ae0 con 0x7f28940b4700 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 10 monclient: _finish_auth 0 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 10 monclient: _check_auth_tickets 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 1 -- 172.21.15.190:0/3865403473 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2890001690 con 0x7f28940b4700 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f289c23e640 5 monclient: authenticate success, global_id 4117 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f289c23e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f2898fd8640 10 monclient: dump: 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:18.294 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:18.294 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:18.294 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:18.294 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.294 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.291+0000 7f289c23e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:18.301 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:18.300+0000 7f761b873040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:25:18.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.318+0000 7f2898fd8640 1 -- 172.21.15.190:0/3865403473 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28900017f0 con 0x7f28940b4700 2024-03-25T18:25:18.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.318+0000 7f2898fd8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:18.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.318+0000 7f2898fd8640 1 -- 172.21.15.190:0/3865403473 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2890001c40 con 0x7f28940b4700 2024-03-25T18:25:18.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.318+0000 7f289c23e640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:18.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.318+0000 7f289c23e640 4 set_mon_vals no callback set 2024-03-25T18:25:18.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.318+0000 7f2898fd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.318+0000 7f2898fd8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:18.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.318+0000 7f2898fd8640 10 monclient: dump: 2024-03-25T18:25:18.319 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:18.319 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:18.319 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.319 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.320 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.320 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:18.320 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:18.320 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:18.320 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:18.320 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.322 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.318+0000 7f289c23e640 10 monclient: shutdown 2024-03-25T18:25:18.322 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.318+0000 7f289c23e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:18.322 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.318+0000 7f289c23e640 1 -- 172.21.15.190:0/3865403473 >> v1:172.21.15.190:6789/0 conn(0x7f28940b4700 legacy=0x7f2894142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.322 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.319+0000 7f289c23e640 1 -- 172.21.15.190:0/3865403473 shutdown_connections 2024-03-25T18:25:18.322 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.319+0000 7f289c23e640 1 -- 172.21.15.190:0/3865403473 >> 172.21.15.190:0/3865403473 conn(0x7f28940668e0 msgr2=0x7f2894066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:18.322 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.319+0000 7f289c23e640 1 -- 172.21.15.190:0/3865403473 shutdown_connections 2024-03-25T18:25:18.322 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.319+0000 7f289c23e640 1 -- 172.21.15.190:0/3865403473 wait complete. 2024-03-25T18:25:18.322 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.320+0000 7f289c23e640 10 monclient: build_initial_monmap 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.320+0000 7f289c23e640 10 monclient: monmap: 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.320+0000 7f289c23e640 1 librados: starting msgr at 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.320+0000 7f289c23e640 1 librados: starting objecter 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.320+0000 7f289c23e640 1 Processor -- start 2024-03-25T18:25:18.323 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.320+0000 7f289c23e640 1 -- start start 2024-03-25T18:25:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.320+0000 7f289c23e640 1 librados: setting wanted keys 2024-03-25T18:25:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.320+0000 7f289c23e640 1 librados: calling monclient init 2024-03-25T18:25:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.320+0000 7f289c23e640 10 monclient: init 2024-03-25T18:25:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.321+0000 7f289c23e640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.321+0000 7f289c23e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.321+0000 7f289c23e640 10 monclient(hunting): picked mon.noname-c con 0x7f28940a96b0 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.321+0000 7f289c23e640 10 monclient(hunting): picked mon.noname-b con 0x7f28940b4700 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.321+0000 7f289c23e640 10 monclient(hunting): picked mon.noname-a con 0x7f2894142a20 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.321+0000 7f289c23e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28941db610 con 0x7f2894142a20 2024-03-25T18:25:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.321+0000 7f289c23e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2894139b60 con 0x7f28940b4700 2024-03-25T18:25:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.321+0000 7f289c23e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28941da9e0 con 0x7f28940a96b0 2024-03-25T18:25:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.321+0000 7f289c23e640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.321+0000 7f289c23e640 10 monclient(hunting): authenticate will time out at 7473.665976s 2024-03-25T18:25:18.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.322+0000 7f2899fda640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f28940a96b0 0x7f28941daf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.190:50858/0 (socket says 172.21.15.190:50858) 2024-03-25T18:25:18.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.322+0000 7f2899fda640 1 -- 172.21.15.190:0/2285446111 learned_addr learned my addr 172.21.15.190:0/2285446111 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:18.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.322+0000 7f288a7fc640 10 client.?.objecter ms_handle_connect 0x7f28940a96b0 2024-03-25T18:25:18.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.322+0000 7f288a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:18.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.324+0000 7f288a7fc640 10 client.?.objecter ms_handle_connect 0x7f28940b4700 2024-03-25T18:25:18.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.324+0000 7f288a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:18.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.325+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207094586 0 0) 0x7f28941da9e0 con 0x7f28940a96b0 2024-03-25T18:25:18.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.325+0000 7f288a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.325+0000 7f288a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.325+0000 7f288a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.325+0000 7f288a7fc640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:25:18.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.325+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2878004eb0 con 0x7f28940a96b0 2024-03-25T18:25:18.327 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.326+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441072064 0 0) 0x7f2894139b60 con 0x7f28940b4700 2024-03-25T18:25:18.327 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.326+0000 7f288a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.327 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.326+0000 7f288a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.327 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.326+0000 7f288a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.327 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.326+0000 7f288a7fc640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:25:18.327 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.326+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28941da9e0 con 0x7f28940b4700 2024-03-25T18:25:18.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.326+0000 7f288a7fc640 10 client.?.objecter ms_handle_connect 0x7f2894142a20 2024-03-25T18:25:18.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.326+0000 7f288a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:18.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.328+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497086855 0 0) 0x7f28941db610 con 0x7f2894142a20 2024-03-25T18:25:18.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.328+0000 7f288a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.328+0000 7f288a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.330 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.328+0000 7f288a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.330 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.328+0000 7f288a7fc640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:25:18.330 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.328+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2894139b60 con 0x7f2894142a20 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.330+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 407615159 0 0) 0x7f2878004eb0 con 0x7f28940a96b0 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.330+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28941db610 con 0x7f28940a96b0 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.330+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2890001c00 con 0x7f28940a96b0 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.330+0000 7f288a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.330+0000 7f288a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.330+0000 7f288a7fc640 10 monclient(hunting): dump: 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:18.332 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.330+0000 7f288a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.331+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3567648795 0 0) 0x7f28941da9e0 con 0x7f28940b4700 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.331+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2878004eb0 con 0x7f28940b4700 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.331+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f28840016b0 con 0x7f28940b4700 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.331+0000 7f288a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.331+0000 7f288a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.331+0000 7f288a7fc640 10 monclient(hunting): dump: 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:18.333 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:18.334 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:18.334 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.335 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.334+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2788241520 0 0) 0x7f2894139b60 con 0x7f2894142a20 2024-03-25T18:25:18.335 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.334+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28941da9e0 con 0x7f2894142a20 2024-03-25T18:25:18.335 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.334+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f288c0019d0 con 0x7f2894142a20 2024-03-25T18:25:18.335 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.334+0000 7f288a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.335 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.334+0000 7f288a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:18.336 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.334+0000 7f288a7fc640 10 monclient(hunting): dump: 2024-03-25T18:25:18.336 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:18.336 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:18.336 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.336 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.336 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.336 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:18.336 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:18.336 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:18.336 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:18.336 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4189470084 0 0) 0x7f2878004eb0 con 0x7f28940b4700 2024-03-25T18:25:18.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 >> v1:172.21.15.99:6790/0 conn(0x7f28940a96b0 legacy=0x7f28941daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 >> v1:172.21.15.99:6789/0 conn(0x7f2894142a20 legacy=0x7f289413c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f288a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:18.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f288a7fc640 1 monclient: found mon.b 2024-03-25T18:25:18.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f288a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:18.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f288a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:18.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28941da6e0 con 0x7f28940b4700 2024-03-25T18:25:18.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f288a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:18.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f288a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:18.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f288a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:18.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f289c23e640 5 monclient: authenticate success, global_id 4120 2024-03-25T18:25:18.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f289c23e640 10 monclient: _renew_subs 2024-03-25T18:25:18.342 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f289c23e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:18.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f289c23e640 1 -- 172.21.15.190:0/2285446111 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2894138160 con 0x7f28940b4700 2024-03-25T18:25:18.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f289c23e640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:18.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f289c23e640 10 monclient: _renew_subs 2024-03-25T18:25:18.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f289c23e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:18.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f289c23e640 1 -- 172.21.15.190:0/2285446111 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f289413ca90 con 0x7f28940b4700 2024-03-25T18:25:18.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.340+0000 7f289c23e640 1 librados: init done 2024-03-25T18:25:18.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.341+0000 7f289c23e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:18.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.341+0000 7f289c23e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:18.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.341+0000 7f289c23e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:18.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.341+0000 7f289c23e640 1 -- 172.21.15.190:0/2285446111 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28941dab70 con 0x7f28940b4700 2024-03-25T18:25:18.358 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:18.357+0000 7f761b873040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:25:18.364 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.363+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28840047e0 con 0x7f28940b4700 2024-03-25T18:25:18.364 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.363+0000 7f288a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:18.364 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.363+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2884004af0 con 0x7f28940b4700 2024-03-25T18:25:18.365 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.363+0000 7f288a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.365 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.363+0000 7f288a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:18.365 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.363+0000 7f288a7fc640 10 monclient: dump: 2024-03-25T18:25:18.365 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:18.365 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:18.365 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.365 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.365 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.365 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:18.365 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:18.365 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:18.365 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:18.365 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.365 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.363+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2884004fe0 con 0x7f28940b4700 2024-03-25T18:25:18.366 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.363+0000 7f288bfff640 4 set_mon_vals no callback set 2024-03-25T18:25:18.366 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.363+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1419314622 0 0) 0x7f2884006470 con 0x7f28940b4700 2024-03-25T18:25:18.366 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.363+0000 7f288a7fc640 10 client.4120.objecter ms_dispatch 0x7f2894000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:25:18.366 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.363+0000 7f288a7fc640 3 client.4120.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:25:18.366 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.363+0000 7f288a7fc640 3 client.4120.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:25:18.366 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.363+0000 7f288a7fc640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-25T18:25:18.385 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:18.384+0000 7ff11184b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:25:18.402 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:18.402+0000 7f761b873040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:25:18.439 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:18.438+0000 7ff11184b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:25:18.483 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:18.482+0000 7ff11184b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:25:18.521 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:18.520+0000 7f761b873040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:25:18.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.534+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2884006870 con 0x7f28940b4700 2024-03-25T18:25:18.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.534+0000 7f288a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:18.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.534+0000 7f288a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:18.536 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:18.535+0000 7ff11184b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:25:18.578 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:18.577+0000 7ff11184b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:25:18.592 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:18.591+0000 7f761b873040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:25:18.606 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.605+0000 7f289c23e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b62a6f41-2c55-459e-b9f1-b90fc256eaea", "id": 0}] 2024-03-25T18:25:18.606 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.605+0000 7f289c23e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b62a6f41-2c55-459e-b9f1-b90fc256eaea", "id": 0}] 2024-03-25T18:25:18.606 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.605+0000 7f289c23e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:18.606 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.605+0000 7f289c23e640 1 -- 172.21.15.190:0/2285446111 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b62a6f41-2c55-459e-b9f1-b90fc256eaea", "id": 0} v 0) v1 -- 0x7f289404f730 con 0x7f28940b4700 2024-03-25T18:25:18.640 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:18.639+0000 7f761b873040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:25:18.693 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:18.692+0000 7f761b873040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:25:18.694 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:18.693+0000 7ff11184b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:25:18.740 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:18.739+0000 7f761b873040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:25:18.770 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:18.769+0000 7ff11184b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:25:18.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.793+0000 7f288a7fc640 1 -- 172.21.15.190:0/2285446111 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b62a6f41-2c55-459e-b9f1-b90fc256eaea", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 612738359 0 732707515) 0x7f2884006ab0 con 0x7f28940b4700 2024-03-25T18:25:18.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.793+0000 7f288a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b62a6f41-2c55-459e-b9f1-b90fc256eaea", "id": 0}] 2024-03-25T18:25:18.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.793+0000 7f288a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:18.795 INFO:teuthology.orchestra.run.smithi190.stdout:0 2024-03-25T18:25:18.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.794+0000 7f289c23e640 10 librados: watch_flush enter 2024-03-25T18:25:18.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.794+0000 7f289c23e640 10 librados: watch_flush exit 2024-03-25T18:25:18.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.794+0000 7f289c23e640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-25T18:25:18.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.794+0000 7f289c23e640 10 client.4120.objecter successfully canceled tick 2024-03-25T18:25:18.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.794+0000 7f289c23e640 10 monclient: shutdown 2024-03-25T18:25:18.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.794+0000 7f289c23e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:18.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.794+0000 7f289c23e640 1 -- 172.21.15.190:0/2285446111 >> v1:172.21.15.190:6789/0 conn(0x7f28940b4700 legacy=0x7f2894137a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.794+0000 7f289c23e640 1 -- 172.21.15.190:0/2285446111 shutdown_connections 2024-03-25T18:25:18.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.794+0000 7f289c23e640 1 -- 172.21.15.190:0/2285446111 >> 172.21.15.190:0/2285446111 conn(0x7f28940668e0 msgr2=0x7f28940b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:18.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.794+0000 7f289c23e640 1 -- 172.21.15.190:0/2285446111 shutdown_connections 2024-03-25T18:25:18.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.794+0000 7f289c23e640 1 -- 172.21.15.190:0/2285446111 wait complete. 2024-03-25T18:25:18.797 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.794+0000 7f289c23e640 1 librados: shutdown 2024-03-25T18:25:18.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new f3b43bbc-3e21-4416-9c71-e9489aa1855a 1 2024-03-25T18:25:18.817 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:18.816+0000 7ff11184b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:25:18.868 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:18.868+0000 7ff11184b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:25:18.915 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:18.914+0000 7ff11184b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:25:18.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.945+0000 7fec93093640 10 monclient: get_monmap_and_config 2024-03-25T18:25:18.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.945+0000 7fec93093640 10 monclient: build_initial_monmap 2024-03-25T18:25:18.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.945+0000 7fec93093640 10 monclient: monmap: 2024-03-25T18:25:18.946 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:18.946 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:18.946 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:25:18.946127+0000 2024-03-25T18:25:18.946 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:25:18.946127+0000 2024-03-25T18:25:18.946 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:18.946 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:18.946 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:18.946 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:25:18.946 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:25:18.947 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.945+0000 7fec93093640 1 Processor -- start 2024-03-25T18:25:18.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.945+0000 7fec93093640 1 -- start start 2024-03-25T18:25:18.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.945+0000 7fec93093640 10 monclient: init 2024-03-25T18:25:18.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.946+0000 7fec93093640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:18.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.946+0000 7fec93093640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:18.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.946+0000 7fec93093640 10 monclient(hunting): picked mon.noname-c con 0x7fec8c0b4700 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:18.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.946+0000 7fec93093640 10 monclient(hunting): picked mon.noname-b con 0x7fec8c142a20 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:18.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.946+0000 7fec93093640 10 monclient(hunting): picked mon.noname-a con 0x7fec8c0a96b0 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:18.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.946+0000 7fec93093640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8c0a9d60 con 0x7fec8c0a96b0 2024-03-25T18:25:18.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.946+0000 7fec93093640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8c13ff10 con 0x7fec8c0b4700 2024-03-25T18:25:18.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.946+0000 7fec93093640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8c0aa1f0 con 0x7fec8c142a20 2024-03-25T18:25:18.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.946+0000 7fec93093640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:18.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.946+0000 7fec93093640 10 monclient(hunting): authenticate will time out at 7474.291151s 2024-03-25T18:25:18.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.947+0000 7fec91630640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fec8c0a96b0 0x7fec8c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.190:41340/0 (socket says 172.21.15.190:41340) 2024-03-25T18:25:18.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.947+0000 7fec91630640 1 -- 172.21.15.190:0/3289262708 learned_addr learned my addr 172.21.15.190:0/3289262708 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:18.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.949+0000 7fec8b7fe640 1 -- 172.21.15.190:0/3289262708 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832022601 0 0) 0x7fec8c0aa1f0 con 0x7fec8c142a20 2024-03-25T18:25:18.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.949+0000 7fec8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.949+0000 7fec8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.949+0000 7fec8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.949+0000 7fec8b7fe640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:25:18.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.949+0000 7fec8b7fe640 1 -- 172.21.15.190:0/3289262708 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec7c003520 con 0x7fec8c142a20 2024-03-25T18:25:18.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.949+0000 7fec8b7fe640 1 -- 172.21.15.190:0/3289262708 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018162767 0 0) 0x7fec8c0a9d60 con 0x7fec8c0a96b0 2024-03-25T18:25:18.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.949+0000 7fec8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.949+0000 7fec8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.949+0000 7fec8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.949+0000 7fec8b7fe640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:25:18.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.949+0000 7fec8b7fe640 1 -- 172.21.15.190:0/3289262708 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec8c0aa1f0 con 0x7fec8c0a96b0 2024-03-25T18:25:18.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.950+0000 7fec8b7fe640 1 -- 172.21.15.190:0/3289262708 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885114747 0 0) 0x7fec8c13ff10 con 0x7fec8c0b4700 2024-03-25T18:25:18.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.950+0000 7fec8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.950+0000 7fec8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.950+0000 7fec8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.950+0000 7fec8b7fe640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:25:18.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.950+0000 7fec8b7fe640 1 -- 172.21.15.190:0/3289262708 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec8c0a9d60 con 0x7fec8c0b4700 2024-03-25T18:25:18.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.952+0000 7fec8b7fe640 1 -- 172.21.15.190:0/3289262708 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4089887765 0 0) 0x7fec8c0aa1f0 con 0x7fec8c0a96b0 2024-03-25T18:25:18.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.952+0000 7fec8b7fe640 1 -- 172.21.15.190:0/3289262708 >> v1:172.21.15.99:6790/0 conn(0x7fec8c142a20 legacy=0x7fec8c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec8b7fe640 1 -- 172.21.15.190:0/3289262708 >> v1:172.21.15.190:6789/0 conn(0x7fec8c0b4700 legacy=0x7fec8c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec8b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:18.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec8b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:25:18.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:18.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec8b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:25:18.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec8b7fe640 1 -- 172.21.15.190:0/3289262708 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8c0b4ae0 con 0x7fec8c0a96b0 2024-03-25T18:25:18.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec8b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:18.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec8b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:18.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:18.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec93093640 5 monclient: authenticate success, global_id 4122 2024-03-25T18:25:18.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec8b7fe640 1 -- 172.21.15.190:0/3289262708 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fec840019f0 con 0x7fec8c0a96b0 2024-03-25T18:25:18.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec8b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:18.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec8b7fe640 10 monclient: dump: 2024-03-25T18:25:18.955 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:18.955 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:18.955 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.955 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.955 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.955 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:18.955 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:18.955 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:18.955 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:18.955 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.953+0000 7fec93093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:18.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.955+0000 7fec8b7fe640 1 -- 172.21.15.190:0/3289262708 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec84001b50 con 0x7fec8c0a96b0 2024-03-25T18:25:18.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.955+0000 7fec8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:18.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.955+0000 7fec8b7fe640 1 -- 172.21.15.190:0/3289262708 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fec84001fa0 con 0x7fec8c0a96b0 2024-03-25T18:25:18.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.955+0000 7fec8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.955+0000 7fec8b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:18.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.955+0000 7fec8b7fe640 10 monclient: dump: 2024-03-25T18:25:18.956 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:18.956 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:18.956 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.956 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.957 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.957 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:18.957 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:18.957 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:18.957 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:18.957 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.955+0000 7fec93093640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:18.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.955+0000 7fec93093640 4 set_mon_vals no callback set 2024-03-25T18:25:18.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.955+0000 7fec93093640 10 monclient: shutdown 2024-03-25T18:25:18.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.955+0000 7fec93093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:18.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.955+0000 7fec93093640 1 -- 172.21.15.190:0/3289262708 >> v1:172.21.15.99:6789/0 conn(0x7fec8c0a96b0 legacy=0x7fec8c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.956+0000 7fec93093640 1 -- 172.21.15.190:0/3289262708 shutdown_connections 2024-03-25T18:25:18.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.956+0000 7fec93093640 1 -- 172.21.15.190:0/3289262708 >> 172.21.15.190:0/3289262708 conn(0x7fec8c0668e0 msgr2=0x7fec8c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:18.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.956+0000 7fec93093640 1 -- 172.21.15.190:0/3289262708 shutdown_connections 2024-03-25T18:25:18.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.956+0000 7fec93093640 1 -- 172.21.15.190:0/3289262708 wait complete. 2024-03-25T18:25:18.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.956+0000 7fec93093640 10 monclient: build_initial_monmap 2024-03-25T18:25:18.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.956+0000 7fec93093640 10 monclient: monmap: 2024-03-25T18:25:18.960 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:18.960 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:18.960 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-25T18:25:18.960 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-25T18:25:18.960 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:18.960 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-25T18:25:18.960 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:18.960 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:18.961 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:18.961 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.956+0000 7fec93093640 1 librados: starting msgr at 2024-03-25T18:25:18.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.956+0000 7fec93093640 1 librados: starting objecter 2024-03-25T18:25:18.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 1 Processor -- start 2024-03-25T18:25:18.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 1 -- start start 2024-03-25T18:25:18.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 1 librados: setting wanted keys 2024-03-25T18:25:18.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 1 librados: calling monclient init 2024-03-25T18:25:18.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 10 monclient: init 2024-03-25T18:25:18.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:18.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:18.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 10 monclient(hunting): picked mon.noname-b con 0x7fec8c0a96b0 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:18.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 10 monclient(hunting): picked mon.noname-c con 0x7fec8c0b4700 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:18.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 10 monclient(hunting): picked mon.noname-a con 0x7fec8c142a20 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:18.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8c13a200 con 0x7fec8c142a20 2024-03-25T18:25:18.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8c0abdb0 con 0x7fec8c0a96b0 2024-03-25T18:25:18.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8c13b880 con 0x7fec8c0b4700 2024-03-25T18:25:18.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:18.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.957+0000 7fec93093640 10 monclient(hunting): authenticate will time out at 7474.302513s 2024-03-25T18:25:18.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.958+0000 7fec91630640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fec8c142a20 0x7fec8c1e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.190:41350/0 (socket says 172.21.15.190:41350) 2024-03-25T18:25:18.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.958+0000 7fec91630640 1 -- 172.21.15.190:0/1993654392 learned_addr learned my addr 172.21.15.190:0/1993654392 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:18.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.958+0000 7fec897fa640 10 client.?.objecter ms_handle_connect 0x7fec8c142a20 2024-03-25T18:25:18.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.958+0000 7fec897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:18.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.958+0000 7fec897fa640 10 client.?.objecter ms_handle_connect 0x7fec8c0a96b0 2024-03-25T18:25:18.963 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.958+0000 7fec897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:18.963 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.958+0000 7fec897fa640 10 client.?.objecter ms_handle_connect 0x7fec8c0b4700 2024-03-25T18:25:18.963 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.958+0000 7fec897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:18.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.960+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536397409 0 0) 0x7fec8c13a200 con 0x7fec8c142a20 2024-03-25T18:25:18.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.960+0000 7fec897fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.960+0000 7fec897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.960+0000 7fec897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.960+0000 7fec897fa640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:25:18.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.960+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec70004eb0 con 0x7fec8c142a20 2024-03-25T18:25:18.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.960+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315733703 0 0) 0x7fec8c13b880 con 0x7fec8c0b4700 2024-03-25T18:25:18.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.960+0000 7fec897fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.960+0000 7fec897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.960+0000 7fec897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.960+0000 7fec897fa640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:25:18.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.960+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec8c13a200 con 0x7fec8c0b4700 2024-03-25T18:25:18.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.961+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159733205 0 0) 0x7fec8c0abdb0 con 0x7fec8c0a96b0 2024-03-25T18:25:18.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.961+0000 7fec897fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:18.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.961+0000 7fec897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:18.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.961+0000 7fec897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:18.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.961+0000 7fec897fa640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:25:18.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.961+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec8c13b880 con 0x7fec8c0a96b0 2024-03-25T18:25:18.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.965+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2239135487 0 0) 0x7fec70004eb0 con 0x7fec8c142a20 2024-03-25T18:25:18.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.965+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec8c0abdb0 con 0x7fec8c142a20 2024-03-25T18:25:18.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.965+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fec84001f60 con 0x7fec8c142a20 2024-03-25T18:25:18.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.965+0000 7fec897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.965+0000 7fec897fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:18.967 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.965+0000 7fec897fa640 10 monclient(hunting): dump: 2024-03-25T18:25:18.967 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:18.967 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:18.967 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.968 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.968 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.968 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:18.968 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:18.968 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:18.968 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:18.968 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.965+0000 7fec897fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:18.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.965+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2507704582 0 0) 0x7fec8c13a200 con 0x7fec8c0b4700 2024-03-25T18:25:18.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.965+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec70004eb0 con 0x7fec8c0b4700 2024-03-25T18:25:18.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.965+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fec780012f0 con 0x7fec8c0b4700 2024-03-25T18:25:18.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.965+0000 7fec897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.965+0000 7fec897fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:18.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.965+0000 7fec897fa640 10 monclient(hunting): dump: 2024-03-25T18:25:18.969 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:18.969 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:18.969 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.969 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.969 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.969 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:18.969 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:18.969 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:18.969 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:18.969 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.967+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2156717864 0 0) 0x7fec8c13b880 con 0x7fec8c0a96b0 2024-03-25T18:25:18.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.967+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec8c13a200 con 0x7fec8c0a96b0 2024-03-25T18:25:18.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.967+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fec80001830 con 0x7fec8c0a96b0 2024-03-25T18:25:18.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.967+0000 7fec897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.967+0000 7fec897fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:18.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.967+0000 7fec897fa640 10 monclient(hunting): dump: 2024-03-25T18:25:18.971 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 827711670 0 0) 0x7fec8c0abdb0 con 0x7fec8c142a20 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 >> v1:172.21.15.99:6790/0 conn(0x7fec8c0b4700 legacy=0x7fec8c138ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 >> v1:172.21.15.190:6789/0 conn(0x7fec8c0a96b0 legacy=0x7fec8c13c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec897fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec897fa640 1 monclient: found mon.a 2024-03-25T18:25:18.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec897fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8c0ac020 con 0x7fec8c142a20 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec897fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec897fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec93093640 5 monclient: authenticate success, global_id 4125 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec93093640 10 monclient: _renew_subs 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec93093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec93093640 1 -- 172.21.15.190:0/1993654392 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8c0ac300 con 0x7fec8c142a20 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec93093640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec93093640 10 monclient: _renew_subs 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec93093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec93093640 1 -- 172.21.15.190:0/1993654392 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8c14e8f0 con 0x7fec8c142a20 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.969+0000 7fec93093640 1 librados: init done 2024-03-25T18:25:18.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.970+0000 7fec93093640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:18.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.970+0000 7fec93093640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:18.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.970+0000 7fec93093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:18.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.970+0000 7fec93093640 1 -- 172.21.15.190:0/1993654392 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8c04f730 con 0x7fec8c142a20 2024-03-25T18:25:18.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.987+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec84004e10 con 0x7fec8c142a20 2024-03-25T18:25:18.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.987+0000 7fec897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:18.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.987+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fec84005120 con 0x7fec8c142a20 2024-03-25T18:25:18.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.987+0000 7fec897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:18.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.987+0000 7fec897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:18.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.987+0000 7fec897fa640 10 monclient: dump: 2024-03-25T18:25:18.989 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:18.989 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:18.989 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.989 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:18.989 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:18.989 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:18.989 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:18.989 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:18.990 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:18.990 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:18.990 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.987+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fec84005610 con 0x7fec8c142a20 2024-03-25T18:25:18.990 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.987+0000 7fec8affd640 4 set_mon_vals no callback set 2024-03-25T18:25:18.990 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.987+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 4099581099 0 0) 0x7fec84006990 con 0x7fec8c142a20 2024-03-25T18:25:18.990 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.987+0000 7fec897fa640 10 client.4125.objecter ms_dispatch 0x7fec8c000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:25:18.990 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.987+0000 7fec897fa640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:25:18.990 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.987+0000 7fec897fa640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:25:18.990 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:18.987+0000 7fec897fa640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T18:25:19.136 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:19.135+0000 7f761b873040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:25:19.150 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.149+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fec840058c0 con 0x7fec8c142a20 2024-03-25T18:25:19.150 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.149+0000 7fec897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:19.150 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.149+0000 7fec897fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:19.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.223+0000 7fec93093640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f3b43bbc-3e21-4416-9c71-e9489aa1855a", "id": 1}] 2024-03-25T18:25:19.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.223+0000 7fec93093640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f3b43bbc-3e21-4416-9c71-e9489aa1855a", "id": 1}] 2024-03-25T18:25:19.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.223+0000 7fec93093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:19.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.223+0000 7fec93093640 1 -- 172.21.15.190:0/1993654392 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f3b43bbc-3e21-4416-9c71-e9489aa1855a", "id": 1} v 0) v1 -- 0x7fec8c0b77d0 con 0x7fec8c142a20 2024-03-25T18:25:19.286 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:19.285+0000 7f761b873040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:25:19.317 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:19.316+0000 7ff11184b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:25:19.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.341+0000 7fec897fa640 1 -- 172.21.15.190:0/1993654392 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f3b43bbc-3e21-4416-9c71-e9489aa1855a", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 816102369 0 940483276) 0x7fec84005ae0 con 0x7fec8c142a20 2024-03-25T18:25:19.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.341+0000 7fec897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f3b43bbc-3e21-4416-9c71-e9489aa1855a", "id": 1}] 2024-03-25T18:25:19.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.341+0000 7fec897fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:19.343 INFO:teuthology.orchestra.run.smithi190.stdout:1 2024-03-25T18:25:19.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.342+0000 7fec93093640 10 librados: watch_flush enter 2024-03-25T18:25:19.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.342+0000 7fec93093640 10 librados: watch_flush exit 2024-03-25T18:25:19.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.342+0000 7fec93093640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T18:25:19.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.342+0000 7fec93093640 10 client.4125.objecter successfully canceled tick 2024-03-25T18:25:19.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.342+0000 7fec93093640 10 monclient: shutdown 2024-03-25T18:25:19.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.342+0000 7fec93093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:19.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.342+0000 7fec93093640 1 -- 172.21.15.190:0/1993654392 >> v1:172.21.15.99:6789/0 conn(0x7fec8c142a20 legacy=0x7fec8c1e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.342+0000 7fec93093640 1 -- 172.21.15.190:0/1993654392 shutdown_connections 2024-03-25T18:25:19.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.342+0000 7fec93093640 1 -- 172.21.15.190:0/1993654392 >> 172.21.15.190:0/1993654392 conn(0x7fec8c0668e0 msgr2=0x7fec8c0b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:19.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.343+0000 7fec93093640 1 -- 172.21.15.190:0/1993654392 shutdown_connections 2024-03-25T18:25:19.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.343+0000 7fec93093640 1 -- 172.21.15.190:0/1993654392 wait complete. 2024-03-25T18:25:19.347 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.343+0000 7fec93093640 1 librados: shutdown 2024-03-25T18:25:19.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 78a0d919-e9cb-4e29-8d0b-aceea8d3a59a 2 2024-03-25T18:25:19.409 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:19.408+0000 7f761b873040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:25:19.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.453+0000 7fa1a86c5640 10 monclient: get_monmap_and_config 2024-03-25T18:25:19.454 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.453+0000 7fa1a86c5640 10 monclient: build_initial_monmap 2024-03-25T18:25:19.454 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.453+0000 7fa1a86c5640 10 monclient: monmap: 2024-03-25T18:25:19.454 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:19.454 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:19.454 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:25:19.454146+0000 2024-03-25T18:25:19.454 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:25:19.454146+0000 2024-03-25T18:25:19.454 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:19.454 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:19.454 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:19.454 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:25:19.454 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:25:19.454 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.455 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:19.453+0000 7ff11184b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:25:19.456 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.453+0000 7fa1a86c5640 1 Processor -- start 2024-03-25T18:25:19.456 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.453+0000 7fa1a86c5640 1 -- start start 2024-03-25T18:25:19.456 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.453+0000 7fa1a86c5640 10 monclient: init 2024-03-25T18:25:19.456 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.454+0000 7fa1a86c5640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:19.457 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.454+0000 7fa1a86c5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:19.457 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.454+0000 7fa1a86c5640 10 monclient(hunting): picked mon.noname-b con 0x7fa1a00b4700 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:19.457 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.454+0000 7fa1a86c5640 10 monclient(hunting): picked mon.noname-c con 0x7fa1a0142a20 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:19.457 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.454+0000 7fa1a86c5640 10 monclient(hunting): picked mon.noname-a con 0x7fa1a00a96b0 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:19.457 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.454+0000 7fa1a86c5640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a00a9d60 con 0x7fa1a00a96b0 2024-03-25T18:25:19.457 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.454+0000 7fa1a86c5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a013ff10 con 0x7fa1a0142a20 2024-03-25T18:25:19.457 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.454+0000 7fa1a86c5640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a00aa1f0 con 0x7fa1a00b4700 2024-03-25T18:25:19.457 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.454+0000 7fa1a86c5640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:19.457 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.454+0000 7fa1a86c5640 10 monclient(hunting): authenticate will time out at 7474.799183s 2024-03-25T18:25:19.457 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.454+0000 7fa1a6c62640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa1a00a96b0 0x7fa1a013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.190:41362/0 (socket says 172.21.15.190:41362) 2024-03-25T18:25:19.457 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.455+0000 7fa1a6c62640 1 -- 172.21.15.190:0/718964808 learned_addr learned my addr 172.21.15.190:0/718964808 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:19.457 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:19.456+0000 7f761b873040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:25:19.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.457+0000 7fa1a545f640 1 -- 172.21.15.190:0/718964808 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830925690 0 0) 0x7fa1a00a9d60 con 0x7fa1a00a96b0 2024-03-25T18:25:19.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.457+0000 7fa1a545f640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.457+0000 7fa1a545f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.457+0000 7fa1a545f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.457+0000 7fa1a545f640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:25:19.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.457+0000 7fa1a545f640 1 -- 172.21.15.190:0/718964808 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa18c003520 con 0x7fa1a00a96b0 2024-03-25T18:25:19.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.457+0000 7fa1a545f640 1 -- 172.21.15.190:0/718964808 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118853496 0 0) 0x7fa1a013ff10 con 0x7fa1a0142a20 2024-03-25T18:25:19.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.457+0000 7fa1a545f640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.457+0000 7fa1a545f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.457+0000 7fa1a545f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.457+0000 7fa1a545f640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:25:19.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.457+0000 7fa1a545f640 1 -- 172.21.15.190:0/718964808 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a00a9d60 con 0x7fa1a0142a20 2024-03-25T18:25:19.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.458+0000 7fa1a545f640 1 -- 172.21.15.190:0/718964808 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053213226 0 0) 0x7fa1a00aa1f0 con 0x7fa1a00b4700 2024-03-25T18:25:19.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.458+0000 7fa1a545f640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.458+0000 7fa1a545f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.458+0000 7fa1a545f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.458+0000 7fa1a545f640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:25:19.460 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.458+0000 7fa1a545f640 1 -- 172.21.15.190:0/718964808 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a013ff10 con 0x7fa1a00b4700 2024-03-25T18:25:19.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 1 -- 172.21.15.190:0/718964808 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2188665774 0 0) 0x7fa18c003520 con 0x7fa1a00a96b0 2024-03-25T18:25:19.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 1 -- 172.21.15.190:0/718964808 >> v1:172.21.15.99:6790/0 conn(0x7fa1a00b4700 legacy=0x7fa1a0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 1 -- 172.21.15.190:0/718964808 >> v1:172.21.15.190:6789/0 conn(0x7fa1a0142a20 legacy=0x7fa1a013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 10 monclient: _finish_hunting 0 2024-03-25T18:25:19.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 1 monclient: found mon.noname-a 2024-03-25T18:25:19.462 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:19.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:25:19.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 1 -- 172.21.15.190:0/718964808 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a00b4ae0 con 0x7fa1a00a96b0 2024-03-25T18:25:19.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 10 monclient: _finish_auth 0 2024-03-25T18:25:19.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 10 monclient: _check_auth_tickets 2024-03-25T18:25:19.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:19.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 1 -- 172.21.15.190:0/718964808 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa198001a10 con 0x7fa1a00a96b0 2024-03-25T18:25:19.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a86c5640 5 monclient: authenticate success, global_id 4128 2024-03-25T18:25:19.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:19.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a545f640 10 monclient: dump: 2024-03-25T18:25:19.464 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:19.464 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:19.464 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.464 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.464 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.464 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:19.465 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:19.465 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:19.465 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:19.465 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.460+0000 7fa1a86c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:19.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a545f640 1 -- 172.21.15.190:0/718964808 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa198001b70 con 0x7fa1a00a96b0 2024-03-25T18:25:19.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a545f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:19.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a545f640 1 -- 172.21.15.190:0/718964808 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa198001f80 con 0x7fa1a00a96b0 2024-03-25T18:25:19.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a86c5640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:19.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a86c5640 4 set_mon_vals no callback set 2024-03-25T18:25:19.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a545f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a545f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:19.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a545f640 10 monclient: dump: 2024-03-25T18:25:19.470 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:19.470 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:19.470 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.470 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.470 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.470 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:19.470 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:19.470 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:19.470 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:19.470 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a86c5640 10 monclient: shutdown 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a86c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a86c5640 1 -- 172.21.15.190:0/718964808 >> v1:172.21.15.99:6789/0 conn(0x7fa1a00a96b0 legacy=0x7fa1a013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a86c5640 1 -- 172.21.15.190:0/718964808 shutdown_connections 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a86c5640 1 -- 172.21.15.190:0/718964808 >> 172.21.15.190:0/718964808 conn(0x7fa1a00668e0 msgr2=0x7fa1a0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a86c5640 1 -- 172.21.15.190:0/718964808 shutdown_connections 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.462+0000 7fa1a86c5640 1 -- 172.21.15.190:0/718964808 wait complete. 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.463+0000 7fa1a86c5640 10 monclient: build_initial_monmap 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.463+0000 7fa1a86c5640 10 monclient: monmap: 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:19.471 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.463+0000 7fa1a86c5640 1 librados: starting msgr at 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.463+0000 7fa1a86c5640 1 librados: starting objecter 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.463+0000 7fa1a86c5640 1 Processor -- start 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.463+0000 7fa1a86c5640 1 -- start start 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.463+0000 7fa1a86c5640 1 librados: setting wanted keys 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.463+0000 7fa1a86c5640 1 librados: calling monclient init 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.463+0000 7fa1a86c5640 10 monclient: init 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.464+0000 7fa1a86c5640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.464+0000 7fa1a86c5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.464+0000 7fa1a86c5640 10 monclient(hunting): picked mon.noname-c con 0x7fa1a00a96b0 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.464+0000 7fa1a86c5640 10 monclient(hunting): picked mon.noname-a con 0x7fa1a00b4700 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.464+0000 7fa1a86c5640 10 monclient(hunting): picked mon.noname-b con 0x7fa1a0142a20 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.464+0000 7fa1a86c5640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a00aba20 con 0x7fa1a00b4700 2024-03-25T18:25:19.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.464+0000 7fa1a86c5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a00abd00 con 0x7fa1a0142a20 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.464+0000 7fa1a86c5640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a013bd80 con 0x7fa1a00a96b0 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.464+0000 7fa1a86c5640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.464+0000 7fa1a86c5640 10 monclient(hunting): authenticate will time out at 7474.809166s 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.464+0000 7fa1a5c60640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa1a00b4700 0x7fa1a01392c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.190:41374/0 (socket says 172.21.15.190:41374) 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.464+0000 7fa1a5c60640 1 -- 172.21.15.190:0/3522141363 learned_addr learned my addr 172.21.15.190:0/3522141363 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.465+0000 7fa192ffd640 10 client.?.objecter ms_handle_connect 0x7fa1a00b4700 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.465+0000 7fa192ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.465+0000 7fa192ffd640 10 client.?.objecter ms_handle_connect 0x7fa1a0142a20 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.465+0000 7fa192ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.465+0000 7fa192ffd640 10 client.?.objecter ms_handle_connect 0x7fa1a00a96b0 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.465+0000 7fa192ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503719051 0 0) 0x7fa1a00abd00 con 0x7fa1a0142a20 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:25:19.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa184004eb0 con 0x7fa1a0142a20 2024-03-25T18:25:19.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595563620 0 0) 0x7fa1a00aba20 con 0x7fa1a00b4700 2024-03-25T18:25:19.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:25:19.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a00abd00 con 0x7fa1a00b4700 2024-03-25T18:25:19.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198983246 0 0) 0x7fa1a013bd80 con 0x7fa1a00a96b0 2024-03-25T18:25:19.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:25:19.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.467+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a00aba20 con 0x7fa1a00a96b0 2024-03-25T18:25:19.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.471+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2130086684 0 0) 0x7fa1a00abd00 con 0x7fa1a00b4700 2024-03-25T18:25:19.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.471+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1a013bd80 con 0x7fa1a00b4700 2024-03-25T18:25:19.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.471+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa19c0019b0 con 0x7fa1a00b4700 2024-03-25T18:25:19.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.471+0000 7fa192ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.471+0000 7fa192ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:19.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.471+0000 7fa192ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:19.478 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:19.478 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:19.478 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.478 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.478 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.478 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:19.478 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:19.478 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:19.479 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:19.479 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.471+0000 7fa192ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:19.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.471+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4111014353 0 0) 0x7fa184004eb0 con 0x7fa1a0142a20 2024-03-25T18:25:19.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.471+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1a00abd00 con 0x7fa1a0142a20 2024-03-25T18:25:19.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.471+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa198001e80 con 0x7fa1a0142a20 2024-03-25T18:25:19.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.471+0000 7fa192ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.471+0000 7fa192ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:19.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.471+0000 7fa192ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:19.480 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:19.480 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:19.480 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.480 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.480 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.480 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:19.480 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:19.480 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:19.480 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:19.480 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2231513530 0 0) 0x7fa1a00aba20 con 0x7fa1a00a96b0 2024-03-25T18:25:19.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa184004eb0 con 0x7fa1a00a96b0 2024-03-25T18:25:19.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa194001830 con 0x7fa1a00a96b0 2024-03-25T18:25:19.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:19.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:19.481 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:19.481 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:19.481 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.481 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.481 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.481 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:19.481 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:19.481 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:19.481 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:19.481 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3029534392 0 0) 0x7fa1a00abd00 con 0x7fa1a0142a20 2024-03-25T18:25:19.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 >> v1:172.21.15.99:6790/0 conn(0x7fa1a00a96b0 legacy=0x7fa1a00ac120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 >> v1:172.21.15.99:6789/0 conn(0x7fa1a00b4700 legacy=0x7fa1a01392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:19.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 1 monclient: found mon.b 2024-03-25T18:25:19.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:19.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:19.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a013ccb0 con 0x7fa1a0142a20 2024-03-25T18:25:19.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:19.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:19.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa192ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:19.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa1a86c5640 5 monclient: authenticate success, global_id 4132 2024-03-25T18:25:19.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa1a86c5640 10 monclient: _renew_subs 2024-03-25T18:25:19.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa1a86c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:19.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa1a86c5640 1 -- 172.21.15.190:0/3522141363 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a013ac10 con 0x7fa1a0142a20 2024-03-25T18:25:19.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa1a86c5640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:19.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa1a86c5640 10 monclient: _renew_subs 2024-03-25T18:25:19.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa1a86c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:19.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa1a86c5640 1 -- 172.21.15.190:0/3522141363 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a013b8d0 con 0x7fa1a0142a20 2024-03-25T18:25:19.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.474+0000 7fa1a86c5640 1 librados: init done 2024-03-25T18:25:19.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.475+0000 7fa1a86c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:19.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.475+0000 7fa1a86c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:19.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.475+0000 7fa1a86c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:19.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.475+0000 7fa1a86c5640 1 -- 172.21.15.190:0/3522141363 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a013bf10 con 0x7fa1a0142a20 2024-03-25T18:25:19.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.477+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa198002020 con 0x7fa1a0142a20 2024-03-25T18:25:19.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.477+0000 7fa192ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:19.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.477+0000 7fa193fff640 4 set_mon_vals no callback set 2024-03-25T18:25:19.570 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:19.569+0000 7ff11184b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:25:19.573 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:19.572+0000 7f761b873040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:25:19.575 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.573+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa198004fb0 con 0x7fa1a0142a20 2024-03-25T18:25:19.575 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.573+0000 7fa192ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.575 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.573+0000 7fa192ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:19.575 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.573+0000 7fa192ffd640 10 monclient: dump: 2024-03-25T18:25:19.575 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:19.575 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:19.575 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.575 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.575 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.575 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:19.575 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:19.575 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:19.575 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:19.575 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.573+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa1980054a0 con 0x7fa1a0142a20 2024-03-25T18:25:19.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.573+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3867723177 0 0) 0x7fa198006a00 con 0x7fa1a0142a20 2024-03-25T18:25:19.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.573+0000 7fa192ffd640 10 client.4132.objecter ms_dispatch 0x7fa1a0000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:25:19.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.573+0000 7fa192ffd640 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:25:19.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.573+0000 7fa192ffd640 3 client.4132.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:25:19.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.573+0000 7fa192ffd640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T18:25:19.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.573+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa198018dc0 con 0x7fa1a0142a20 2024-03-25T18:25:19.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.573+0000 7fa192ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:19.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.573+0000 7fa192ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:19.616 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:19.615+0000 7ff11184b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:25:19.647 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.646+0000 7fa1a86c5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "78a0d919-e9cb-4e29-8d0b-aceea8d3a59a", "id": 2}] 2024-03-25T18:25:19.647 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.646+0000 7fa1a86c5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "78a0d919-e9cb-4e29-8d0b-aceea8d3a59a", "id": 2}] 2024-03-25T18:25:19.647 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.646+0000 7fa1a86c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:19.647 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.646+0000 7fa1a86c5640 1 -- 172.21.15.190:0/3522141363 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd new", "uuid": "78a0d919-e9cb-4e29-8d0b-aceea8d3a59a", "id": 2} v 0) v1 -- 0x7fa1a013bb80 con 0x7fa1a0142a20 2024-03-25T18:25:19.671 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:19.670+0000 7f761b873040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:25:19.717 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.716+0000 7fa192ffd640 1 -- 172.21.15.190:0/3522141363 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "78a0d919-e9cb-4e29-8d0b-aceea8d3a59a", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3081747444 0 216599125) 0x7fa1980065e0 con 0x7fa1a0142a20 2024-03-25T18:25:19.717 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.716+0000 7fa192ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "78a0d919-e9cb-4e29-8d0b-aceea8d3a59a", "id": 2}] 2024-03-25T18:25:19.717 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.716+0000 7fa192ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:19.717 INFO:teuthology.orchestra.run.smithi190.stdout:2 2024-03-25T18:25:19.718 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.716+0000 7fa1a86c5640 10 librados: watch_flush enter 2024-03-25T18:25:19.718 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.716+0000 7fa1a86c5640 10 librados: watch_flush exit 2024-03-25T18:25:19.718 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.716+0000 7fa1a86c5640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T18:25:19.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.716+0000 7fa1a86c5640 10 client.4132.objecter successfully canceled tick 2024-03-25T18:25:19.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.716+0000 7fa1a86c5640 10 monclient: shutdown 2024-03-25T18:25:19.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.716+0000 7fa1a86c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:19.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.716+0000 7fa1a86c5640 1 -- 172.21.15.190:0/3522141363 >> v1:172.21.15.190:6789/0 conn(0x7fa1a0142a20 legacy=0x7fa1a01e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.717+0000 7fa1a86c5640 1 -- 172.21.15.190:0/3522141363 shutdown_connections 2024-03-25T18:25:19.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.717+0000 7fa1a86c5640 1 -- 172.21.15.190:0/3522141363 >> 172.21.15.190:0/3522141363 conn(0x7fa1a00668e0 msgr2=0x7fa1a013d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:19.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.717+0000 7fa1a86c5640 1 -- 172.21.15.190:0/3522141363 shutdown_connections 2024-03-25T18:25:19.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.717+0000 7fa1a86c5640 1 -- 172.21.15.190:0/3522141363 wait complete. 2024-03-25T18:25:19.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.717+0000 7fa1a86c5640 1 librados: shutdown 2024-03-25T18:25:19.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new b1c2ae73-1010-4c54-93dc-dcac83badf88 3 2024-03-25T18:25:19.733 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:19.732+0000 7ff11184b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:25:19.827 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:19.826+0000 7ff11184b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:25:19.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.827+0000 7ff681e4e640 10 monclient: get_monmap_and_config 2024-03-25T18:25:19.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.827+0000 7ff681e4e640 10 monclient: build_initial_monmap 2024-03-25T18:25:19.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.827+0000 7ff681e4e640 10 monclient: monmap: 2024-03-25T18:25:19.828 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:19.828 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:19.828 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:25:19.828553+0000 2024-03-25T18:25:19.828 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:25:19.828553+0000 2024-03-25T18:25:19.828 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:19.828 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:19.829 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:19.829 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:25:19.829 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:25:19.829 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.828+0000 7ff681e4e640 1 Processor -- start 2024-03-25T18:25:19.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.828+0000 7ff681e4e640 1 -- start start 2024-03-25T18:25:19.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.828+0000 7ff681e4e640 10 monclient: init 2024-03-25T18:25:19.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.828+0000 7ff681e4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:19.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.828+0000 7ff681e4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:19.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.828+0000 7ff681e4e640 10 monclient(hunting): picked mon.noname-a con 0x7ff67c0b4700 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:19.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.828+0000 7ff681e4e640 10 monclient(hunting): picked mon.noname-c con 0x7ff67c142a20 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:19.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.828+0000 7ff681e4e640 10 monclient(hunting): picked mon.noname-b con 0x7ff67c0a96b0 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:19.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.828+0000 7ff681e4e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c0a9d60 con 0x7ff67c0b4700 2024-03-25T18:25:19.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.828+0000 7ff681e4e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c13ff10 con 0x7ff67c142a20 2024-03-25T18:25:19.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.828+0000 7ff681e4e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c0aa1f0 con 0x7ff67c0a96b0 2024-03-25T18:25:19.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.828+0000 7ff681e4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:19.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.828+0000 7ff681e4e640 10 monclient(hunting): authenticate will time out at 7475.173588s 2024-03-25T18:25:19.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.829+0000 7ff67affd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ff67c142a20 0x7ff67c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:47906/0 (socket says 172.21.15.190:47906) 2024-03-25T18:25:19.831 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.829+0000 7ff67affd640 1 -- 172.21.15.190:0/3809912209 learned_addr learned my addr 172.21.15.190:0/3809912209 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.831+0000 7ff67a7fc640 1 -- 172.21.15.190:0/3809912209 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889543161 0 0) 0x7ff67c13ff10 con 0x7ff67c142a20 2024-03-25T18:25:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.831+0000 7ff67a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.831+0000 7ff67a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.831+0000 7ff67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.831+0000 7ff67a7fc640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:25:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.831+0000 7ff67a7fc640 1 -- 172.21.15.190:0/3809912209 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff664003520 con 0x7ff67c142a20 2024-03-25T18:25:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.832+0000 7ff67a7fc640 1 -- 172.21.15.190:0/3809912209 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493487958 0 0) 0x7ff67c0aa1f0 con 0x7ff67c0a96b0 2024-03-25T18:25:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.832+0000 7ff67a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.832+0000 7ff67a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.832+0000 7ff67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.832+0000 7ff67a7fc640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:25:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.832+0000 7ff67a7fc640 1 -- 172.21.15.190:0/3809912209 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff67c13ff10 con 0x7ff67c0a96b0 2024-03-25T18:25:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.832+0000 7ff67a7fc640 1 -- 172.21.15.190:0/3809912209 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376840856 0 0) 0x7ff67c0a9d60 con 0x7ff67c0b4700 2024-03-25T18:25:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.832+0000 7ff67a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.834 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.832+0000 7ff67a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.834 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.832+0000 7ff67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.834 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.832+0000 7ff67a7fc640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:25:19.834 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.832+0000 7ff67a7fc640 1 -- 172.21.15.190:0/3809912209 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff67c0aa1f0 con 0x7ff67c0b4700 2024-03-25T18:25:19.835 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 1 -- 172.21.15.190:0/3809912209 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1137834418 0 0) 0x7ff664003520 con 0x7ff67c142a20 2024-03-25T18:25:19.835 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 1 -- 172.21.15.190:0/3809912209 >> v1:172.21.15.99:6790/0 conn(0x7ff67c0a96b0 legacy=0x7ff67c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 1 -- 172.21.15.190:0/3809912209 >> v1:172.21.15.99:6789/0 conn(0x7ff67c0b4700 legacy=0x7ff67c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 1 -- 172.21.15.190:0/3809912209 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c0b4ae0 con 0x7ff67c142a20 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 1 -- 172.21.15.190:0/3809912209 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff670001690 con 0x7ff67c142a20 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff681e4e640 5 monclient: authenticate success, global_id 4135 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff681e4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:19.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff67a7fc640 10 monclient: dump: 2024-03-25T18:25:19.837 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:19.837 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:19.837 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.837 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.837 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.837 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:19.837 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:19.837 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:19.837 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:19.837 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.837 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.834+0000 7ff681e4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:19.838 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:19.834+0000 7f761b873040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:25:19.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.836+0000 7ff67a7fc640 1 -- 172.21.15.190:0/3809912209 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6700017f0 con 0x7ff67c142a20 2024-03-25T18:25:19.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.836+0000 7ff67a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:19.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.836+0000 7ff67a7fc640 1 -- 172.21.15.190:0/3809912209 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff670001c00 con 0x7ff67c142a20 2024-03-25T18:25:19.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.836+0000 7ff67a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.836+0000 7ff67a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:19.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.836+0000 7ff67a7fc640 10 monclient: dump: 2024-03-25T18:25:19.840 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:19.840 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:19.840 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.840 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.840 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.840 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:19.840 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:19.840 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:19.840 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:19.840 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.840 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.836+0000 7ff681e4e640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:19.840 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.836+0000 7ff681e4e640 4 set_mon_vals no callback set 2024-03-25T18:25:19.840 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.837+0000 7ff681e4e640 10 monclient: shutdown 2024-03-25T18:25:19.840 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.837+0000 7ff681e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.837+0000 7ff681e4e640 1 -- 172.21.15.190:0/3809912209 >> v1:172.21.15.190:6789/0 conn(0x7ff67c142a20 legacy=0x7ff67c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.837+0000 7ff681e4e640 1 -- 172.21.15.190:0/3809912209 shutdown_connections 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.837+0000 7ff681e4e640 1 -- 172.21.15.190:0/3809912209 >> 172.21.15.190:0/3809912209 conn(0x7ff67c0668e0 msgr2=0x7ff67c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.837+0000 7ff681e4e640 1 -- 172.21.15.190:0/3809912209 shutdown_connections 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.837+0000 7ff681e4e640 1 -- 172.21.15.190:0/3809912209 wait complete. 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.838+0000 7ff681e4e640 10 monclient: build_initial_monmap 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.838+0000 7ff681e4e640 10 monclient: monmap: 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:19.841 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:19.842 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.838+0000 7ff681e4e640 1 librados: starting msgr at 2024-03-25T18:25:19.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.838+0000 7ff681e4e640 1 librados: starting objecter 2024-03-25T18:25:19.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.838+0000 7ff681e4e640 1 Processor -- start 2024-03-25T18:25:19.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.838+0000 7ff681e4e640 1 -- start start 2024-03-25T18:25:19.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.838+0000 7ff681e4e640 1 librados: setting wanted keys 2024-03-25T18:25:19.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.838+0000 7ff681e4e640 1 librados: calling monclient init 2024-03-25T18:25:19.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.838+0000 7ff681e4e640 10 monclient: init 2024-03-25T18:25:19.844 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.839+0000 7ff681e4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:19.844 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.839+0000 7ff681e4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:19.844 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.839+0000 7ff681e4e640 10 monclient(hunting): picked mon.noname-c con 0x7ff67c0a96b0 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:19.844 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.839+0000 7ff681e4e640 10 monclient(hunting): picked mon.noname-b con 0x7ff67c0b4700 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:19.844 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.839+0000 7ff681e4e640 10 monclient(hunting): picked mon.noname-a con 0x7ff67c142a20 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:19.844 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.839+0000 7ff681e4e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c0720f0 con 0x7ff67c142a20 2024-03-25T18:25:19.844 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.839+0000 7ff681e4e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c06e780 con 0x7ff67c0b4700 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.839+0000 7ff681e4e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c06e910 con 0x7ff67c0a96b0 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.839+0000 7ff681e4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.839+0000 7ff681e4e640 10 monclient(hunting): authenticate will time out at 7475.184028s 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.839+0000 7ff67affd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ff67c0b4700 0x7ff67c1dc280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:47916/0 (socket says 172.21.15.190:47916) 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.839+0000 7ff67affd640 1 -- 172.21.15.190:0/3967581881 learned_addr learned my addr 172.21.15.190:0/3967581881 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.839+0000 7ff67bfff640 1 --1- 172.21.15.190:0/3967581881 >> v1:172.21.15.99:6789/0 conn(0x7ff67c142a20 0x7ff67c1e63c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.190:41392/0 (socket says 172.21.15.190:41392) 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.840+0000 7ff663fff640 10 client.?.objecter ms_handle_connect 0x7ff67c0b4700 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.840+0000 7ff663fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.840+0000 7ff663fff640 10 client.?.objecter ms_handle_connect 0x7ff67c142a20 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.840+0000 7ff663fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.840+0000 7ff663fff640 10 client.?.objecter ms_handle_connect 0x7ff67c0a96b0 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.840+0000 7ff663fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710602199 0 0) 0x7ff67c06e780 con 0x7ff67c0b4700 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:25:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff668004eb0 con 0x7ff67c0b4700 2024-03-25T18:25:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039018803 0 0) 0x7ff67c06e910 con 0x7ff67c0a96b0 2024-03-25T18:25:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:25:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff67c06e780 con 0x7ff67c0a96b0 2024-03-25T18:25:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200637565 0 0) 0x7ff67c0720f0 con 0x7ff67c142a20 2024-03-25T18:25:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:25:19.847 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.842+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff67c06e910 con 0x7ff67c142a20 2024-03-25T18:25:19.848 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.847+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2285302758 0 0) 0x7ff67c06e910 con 0x7ff67c142a20 2024-03-25T18:25:19.848 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.847+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff67c0720f0 con 0x7ff67c142a20 2024-03-25T18:25:19.848 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.847+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff674001b50 con 0x7ff67c142a20 2024-03-25T18:25:19.848 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.847+0000 7ff663fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.848 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.847+0000 7ff663fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:19.848 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.847+0000 7ff663fff640 10 monclient(hunting): dump: 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.847+0000 7ff663fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.847+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2351794781 0 0) 0x7ff67c06e780 con 0x7ff67c0a96b0 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.847+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff67c06e910 con 0x7ff67c0a96b0 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.847+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff66c001870 con 0x7ff67c0a96b0 2024-03-25T18:25:19.849 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.847+0000 7ff663fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.847+0000 7ff663fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:19.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.847+0000 7ff663fff640 10 monclient(hunting): dump: 2024-03-25T18:25:19.850 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:19.850 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:19.850 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.850 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.850 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.850 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:19.850 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:19.850 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:19.850 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:19.850 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.848+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2401202783 0 0) 0x7ff668004eb0 con 0x7ff67c0b4700 2024-03-25T18:25:19.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.848+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff67c06e780 con 0x7ff67c0b4700 2024-03-25T18:25:19.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.848+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff670001bc0 con 0x7ff67c0b4700 2024-03-25T18:25:19.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.848+0000 7ff663fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.848+0000 7ff663fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:19.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.848+0000 7ff663fff640 10 monclient(hunting): dump: 2024-03-25T18:25:19.851 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:19.851 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:19.851 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.852 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.852 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.852 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:19.852 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:19.852 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:19.852 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:19.852 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2165913026 0 0) 0x7ff67c0720f0 con 0x7ff67c142a20 2024-03-25T18:25:19.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 >> v1:172.21.15.99:6790/0 conn(0x7ff67c0a96b0 legacy=0x7ff67c1dbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 >> v1:172.21.15.190:6789/0 conn(0x7ff67c0b4700 legacy=0x7ff67c1dc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:19.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff663fff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:19.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff663fff640 1 monclient: found mon.a 2024-03-25T18:25:19.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff663fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:19.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff663fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:19.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c136ee0 con 0x7ff67c142a20 2024-03-25T18:25:19.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff663fff640 10 monclient: _finish_auth 0 2024-03-25T18:25:19.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff663fff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:19.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff663fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:19.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff681e4e640 5 monclient: authenticate success, global_id 4137 2024-03-25T18:25:19.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff681e4e640 10 monclient: _renew_subs 2024-03-25T18:25:19.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff681e4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:19.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff681e4e640 1 -- 172.21.15.190:0/3967581881 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff67c137190 con 0x7ff67c142a20 2024-03-25T18:25:19.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff681e4e640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:19.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff681e4e640 10 monclient: _renew_subs 2024-03-25T18:25:19.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff681e4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:19.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff681e4e640 1 -- 172.21.15.190:0/3967581881 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67c0712c0 con 0x7ff67c142a20 2024-03-25T18:25:19.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff681e4e640 1 librados: init done 2024-03-25T18:25:19.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff681e4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:19.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.851+0000 7ff681e4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:19.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.852+0000 7ff681e4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:19.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.852+0000 7ff681e4e640 1 -- 172.21.15.190:0/3967581881 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff67c04f730 con 0x7ff67c142a20 2024-03-25T18:25:19.883 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:19.882+0000 7f761b873040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:25:19.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.949+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff674002000 con 0x7ff67c142a20 2024-03-25T18:25:19.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.949+0000 7ff663fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:19.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.950+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff674003b30 con 0x7ff67c142a20 2024-03-25T18:25:19.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.950+0000 7ff663fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:19.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.950+0000 7ff663fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:19.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.950+0000 7ff663fff640 10 monclient: dump: 2024-03-25T18:25:19.951 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:19.951 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:19.951 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.951 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:19.951 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:19.951 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:19.952 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:19.952 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:19.952 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:19.952 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:19.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.950+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff6740040e0 con 0x7ff67c142a20 2024-03-25T18:25:19.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.950+0000 7ff679ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:19.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.950+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2352812454 0 0) 0x7ff674006800 con 0x7ff67c142a20 2024-03-25T18:25:19.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.950+0000 7ff663fff640 10 client.4137.objecter ms_dispatch 0x7ff67c000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:25:19.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.950+0000 7ff663fff640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:25:19.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.950+0000 7ff663fff640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:25:19.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.950+0000 7ff663fff640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T18:25:19.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.950+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff674004610 con 0x7ff67c142a20 2024-03-25T18:25:19.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.950+0000 7ff663fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:19.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:19.950+0000 7ff663fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:19.988 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:19.987+0000 7ff11184b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:25:20.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.023+0000 7ff681e4e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b1c2ae73-1010-4c54-93dc-dcac83badf88", "id": 3}] 2024-03-25T18:25:20.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.023+0000 7ff681e4e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b1c2ae73-1010-4c54-93dc-dcac83badf88", "id": 3}] 2024-03-25T18:25:20.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.023+0000 7ff681e4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:20.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.023+0000 7ff681e4e640 1 -- 172.21.15.190:0/3967581881 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b1c2ae73-1010-4c54-93dc-dcac83badf88", "id": 3} v 0) v1 -- 0x7ff67c0b7750 con 0x7ff67c142a20 2024-03-25T18:25:20.032 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:20.031+0000 7ff11184b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:25:20.039 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:20.038+0000 7f761b873040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:25:20.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.089+0000 7ff663fff640 1 -- 172.21.15.190:0/3967581881 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b1c2ae73-1010-4c54-93dc-dcac83badf88", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 748593643 0 525046306) 0x7ff674006370 con 0x7ff67c142a20 2024-03-25T18:25:20.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.089+0000 7ff663fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b1c2ae73-1010-4c54-93dc-dcac83badf88", "id": 3}] 2024-03-25T18:25:20.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.089+0000 7ff663fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:20.091 INFO:teuthology.orchestra.run.smithi190.stdout:3 2024-03-25T18:25:20.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.090+0000 7ff681e4e640 10 librados: watch_flush enter 2024-03-25T18:25:20.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.090+0000 7ff681e4e640 10 librados: watch_flush exit 2024-03-25T18:25:20.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.090+0000 7ff681e4e640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T18:25:20.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.090+0000 7ff681e4e640 10 client.4137.objecter successfully canceled tick 2024-03-25T18:25:20.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.090+0000 7ff681e4e640 10 monclient: shutdown 2024-03-25T18:25:20.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.090+0000 7ff681e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:20.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.090+0000 7ff681e4e640 1 -- 172.21.15.190:0/3967581881 >> v1:172.21.15.99:6789/0 conn(0x7ff67c142a20 legacy=0x7ff67c1e63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.090+0000 7ff681e4e640 1 -- 172.21.15.190:0/3967581881 shutdown_connections 2024-03-25T18:25:20.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.090+0000 7ff681e4e640 1 -- 172.21.15.190:0/3967581881 >> 172.21.15.190:0/3967581881 conn(0x7ff67c0668e0 msgr2=0x7ff67c14e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:20.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.090+0000 7ff681e4e640 1 -- 172.21.15.190:0/3967581881 shutdown_connections 2024-03-25T18:25:20.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.090+0000 7ff681e4e640 1 -- 172.21.15.190:0/3967581881 wait complete. 2024-03-25T18:25:20.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.090+0000 7ff681e4e640 1 librados: shutdown 2024-03-25T18:25:20.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 031772d9-ed09-4828-87a7-6354210e6cc6 4 2024-03-25T18:25:20.194 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:20.193+0000 7ff11184b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:25:20.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.200+0000 7f2101315640 10 monclient: get_monmap_and_config 2024-03-25T18:25:20.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.200+0000 7f2101315640 10 monclient: build_initial_monmap 2024-03-25T18:25:20.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.200+0000 7f2101315640 10 monclient: monmap: 2024-03-25T18:25:20.202 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:20.202 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.202 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:25:20.201969+0000 2024-03-25T18:25:20.202 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:25:20.201969+0000 2024-03-25T18:25:20.202 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:20.202 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.202 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:20.202 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:25:20.202 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:25:20.202 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.201+0000 7f2101315640 1 Processor -- start 2024-03-25T18:25:20.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.201+0000 7f2101315640 1 -- start start 2024-03-25T18:25:20.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.201+0000 7f2101315640 10 monclient: init 2024-03-25T18:25:20.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.202+0000 7f2101315640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:20.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.202+0000 7f2101315640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:20.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.202+0000 7f2101315640 10 monclient(hunting): picked mon.noname-a con 0x7f20fc0b4700 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:20.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.202+0000 7f2101315640 10 monclient(hunting): picked mon.noname-b con 0x7f20fc142a20 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:20.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.202+0000 7f2101315640 10 monclient(hunting): picked mon.noname-c con 0x7f20fc0a96b0 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:20.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.202+0000 7f2101315640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc0a9d60 con 0x7f20fc0b4700 2024-03-25T18:25:20.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.202+0000 7f2101315640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc13ff10 con 0x7f20fc0a96b0 2024-03-25T18:25:20.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.202+0000 7f2101315640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc0aa1f0 con 0x7f20fc142a20 2024-03-25T18:25:20.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.202+0000 7f2101315640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:20.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.202+0000 7f2101315640 10 monclient(hunting): authenticate will time out at 7475.547476s 2024-03-25T18:25:20.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.203+0000 7f20f3fff640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f20fc142a20 0x7f20fc13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.190:50912/0 (socket says 172.21.15.190:50912) 2024-03-25T18:25:20.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.203+0000 7f20f3fff640 1 -- 172.21.15.190:0/2028994431 learned_addr learned my addr 172.21.15.190:0/2028994431 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:20.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.209+0000 7f20fa7fc640 1 -- 172.21.15.190:0/2028994431 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334995313 0 0) 0x7f20fc0aa1f0 con 0x7f20fc142a20 2024-03-25T18:25:20.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.209+0000 7f20fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.209+0000 7f20fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.209+0000 7f20fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.209+0000 7f20fa7fc640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:25:20.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.209+0000 7f20fa7fc640 1 -- 172.21.15.190:0/2028994431 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e8003520 con 0x7f20fc142a20 2024-03-25T18:25:20.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.243+0000 7f20fa7fc640 1 -- 172.21.15.190:0/2028994431 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617597922 0 0) 0x7f20fc0a9d60 con 0x7f20fc0b4700 2024-03-25T18:25:20.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.243+0000 7f20fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.243+0000 7f20fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.243+0000 7f20fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.243+0000 7f20fa7fc640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:25:20.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.243+0000 7f20fa7fc640 1 -- 172.21.15.190:0/2028994431 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20fc0aa1f0 con 0x7f20fc0b4700 2024-03-25T18:25:20.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.247+0000 7f20fa7fc640 1 -- 172.21.15.190:0/2028994431 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917715214 0 0) 0x7f20fc13ff10 con 0x7f20fc0a96b0 2024-03-25T18:25:20.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.247+0000 7f20fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.247+0000 7f20fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.247+0000 7f20fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.247+0000 7f20fa7fc640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:25:20.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.247+0000 7f20fa7fc640 1 -- 172.21.15.190:0/2028994431 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20fc0a9d60 con 0x7f20fc0a96b0 2024-03-25T18:25:20.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 1 -- 172.21.15.190:0/2028994431 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3816284311 0 0) 0x7f20e8003520 con 0x7f20fc142a20 2024-03-25T18:25:20.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 1 -- 172.21.15.190:0/2028994431 >> v1:172.21.15.190:6789/0 conn(0x7f20fc0a96b0 legacy=0x7f20fc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 1 -- 172.21.15.190:0/2028994431 >> v1:172.21.15.99:6789/0 conn(0x7f20fc0b4700 legacy=0x7f20fc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:20.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 1 monclient: found mon.noname-b 2024-03-25T18:25:20.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:20.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:25:20.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 1 -- 172.21.15.190:0/2028994431 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20fc0b4ae0 con 0x7f20fc142a20 2024-03-25T18:25:20.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:20.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:20.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:20.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 1 -- 172.21.15.190:0/2028994431 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f20e4001690 con 0x7f20fc142a20 2024-03-25T18:25:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f2101315640 5 monclient: authenticate success, global_id 4139 2024-03-25T18:25:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f2101315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f20fa7fc640 10 monclient: dump: 2024-03-25T18:25:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.251 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.252 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.252 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.248+0000 7f2101315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.254+0000 7f20fa7fc640 1 -- 172.21.15.190:0/2028994431 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20e4001870 con 0x7f20fc142a20 2024-03-25T18:25:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.254+0000 7f20fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.254+0000 7f20fa7fc640 1 -- 172.21.15.190:0/2028994431 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f20e4001c80 con 0x7f20fc142a20 2024-03-25T18:25:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.254+0000 7f20fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.254+0000 7f20fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.254+0000 7f20fa7fc640 10 monclient: dump: 2024-03-25T18:25:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.255 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.256 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.256 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.256 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.256 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.256 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.254+0000 7f2101315640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:20.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.254+0000 7f2101315640 4 set_mon_vals no callback set 2024-03-25T18:25:20.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.254+0000 7f2101315640 10 monclient: shutdown 2024-03-25T18:25:20.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.254+0000 7f2101315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:20.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.254+0000 7f2101315640 1 -- 172.21.15.190:0/2028994431 >> v1:172.21.15.99:6790/0 conn(0x7f20fc142a20 legacy=0x7f20fc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.255+0000 7f2101315640 1 -- 172.21.15.190:0/2028994431 shutdown_connections 2024-03-25T18:25:20.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.255+0000 7f2101315640 1 -- 172.21.15.190:0/2028994431 >> 172.21.15.190:0/2028994431 conn(0x7f20fc0668e0 msgr2=0x7f20fc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:20.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.255+0000 7f2101315640 1 -- 172.21.15.190:0/2028994431 shutdown_connections 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.255+0000 7f2101315640 1 -- 172.21.15.190:0/2028994431 wait complete. 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.255+0000 7f2101315640 10 monclient: build_initial_monmap 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.255+0000 7f2101315640 10 monclient: monmap: 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 1 librados: starting msgr at 2024-03-25T18:25:20.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 1 librados: starting objecter 2024-03-25T18:25:20.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 1 Processor -- start 2024-03-25T18:25:20.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 1 -- start start 2024-03-25T18:25:20.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 1 librados: setting wanted keys 2024-03-25T18:25:20.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 1 librados: calling monclient init 2024-03-25T18:25:20.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 10 monclient: init 2024-03-25T18:25:20.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:20.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:20.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 10 monclient(hunting): picked mon.noname-a con 0x7f20fc0a96b0 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:20.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 10 monclient(hunting): picked mon.noname-c con 0x7f20fc0b4700 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:20.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 10 monclient(hunting): picked mon.noname-b con 0x7f20fc142a20 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:20.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc1dac00 con 0x7f20fc0a96b0 2024-03-25T18:25:20.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc0abf00 con 0x7f20fc142a20 2024-03-25T18:25:20.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc1db830 con 0x7f20fc0b4700 2024-03-25T18:25:20.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:20.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.256+0000 7f2101315640 10 monclient(hunting): authenticate will time out at 7475.601635s 2024-03-25T18:25:20.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.258+0000 7f20faffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f20fc0a96b0 0x7f20fc1db120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.190:41406/0 (socket says 172.21.15.190:41406) 2024-03-25T18:25:20.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.258+0000 7f20faffd640 1 -- 172.21.15.190:0/3174938308 learned_addr learned my addr 172.21.15.190:0/3174938308 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:20.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.258+0000 7f20f37fe640 10 client.?.objecter ms_handle_connect 0x7f20fc0b4700 2024-03-25T18:25:20.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.258+0000 7f20f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:20.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.259+0000 7f20f37fe640 10 client.?.objecter ms_handle_connect 0x7f20fc0a96b0 2024-03-25T18:25:20.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.259+0000 7f20f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:20.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.261+0000 7f20f37fe640 10 client.?.objecter ms_handle_connect 0x7f20fc142a20 2024-03-25T18:25:20.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.261+0000 7f20f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:20.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.262+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208705653 0 0) 0x7f20fc1db830 con 0x7f20fc0b4700 2024-03-25T18:25:20.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.262+0000 7f20f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.262+0000 7f20f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.262+0000 7f20f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.262+0000 7f20f37fe640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:25:20.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.262+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20dc004eb0 con 0x7f20fc0b4700 2024-03-25T18:25:20.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.262+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853136626 0 0) 0x7f20fc1dac00 con 0x7f20fc0a96b0 2024-03-25T18:25:20.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.262+0000 7f20f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.262+0000 7f20f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.262+0000 7f20f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.262+0000 7f20f37fe640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:25:20.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.262+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20fc1db830 con 0x7f20fc0a96b0 2024-03-25T18:25:20.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.263+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825301444 0 0) 0x7f20fc0abf00 con 0x7f20fc142a20 2024-03-25T18:25:20.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.263+0000 7f20f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.263+0000 7f20f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.263+0000 7f20f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.263+0000 7f20f37fe640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:25:20.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.263+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20fc1dac00 con 0x7f20fc142a20 2024-03-25T18:25:20.269 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 819088405 0 0) 0x7f20fc1dac00 con 0x7f20fc142a20 2024-03-25T18:25:20.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20fc0abf00 con 0x7f20fc142a20 2024-03-25T18:25:20.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f20f4001bd0 con 0x7f20fc142a20 2024-03-25T18:25:20.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:20.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 10 monclient(hunting): dump: 2024-03-25T18:25:20.270 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.270 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.270 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.270 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.270 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.271 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.271 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.271 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.271 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.271 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:20.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 245803534 0 0) 0x7f20dc004eb0 con 0x7f20fc0b4700 2024-03-25T18:25:20.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20fc1dac00 con 0x7f20fc0b4700 2024-03-25T18:25:20.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f20e4001b80 con 0x7f20fc0b4700 2024-03-25T18:25:20.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:20.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 10 monclient(hunting): dump: 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 455813507 0 0) 0x7f20fc1db830 con 0x7f20fc0a96b0 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20dc004eb0 con 0x7f20fc0a96b0 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f20ec001970 con 0x7f20fc0a96b0 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.272 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:20.273 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.268+0000 7f20f37fe640 10 monclient(hunting): dump: 2024-03-25T18:25:20.273 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.273 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.273 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.273 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.273 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.273 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.273 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.273 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.273 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.273 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.274 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3721511485 0 0) 0x7f20fc0abf00 con 0x7f20fc142a20 2024-03-25T18:25:20.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 >> v1:172.21.15.99:6790/0 conn(0x7f20fc0b4700 legacy=0x7f20fc1d4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 >> v1:172.21.15.99:6789/0 conn(0x7f20fc0a96b0 legacy=0x7f20fc1db120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f20f37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:20.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f20f37fe640 1 monclient: found mon.b 2024-03-25T18:25:20.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f20f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:20.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f20f37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:20.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20fc1db9c0 con 0x7f20fc142a20 2024-03-25T18:25:20.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f20f37fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:20.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f20f37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:20.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f20f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:20.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f2101315640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:25:20.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f2101315640 10 monclient: _renew_subs 2024-03-25T18:25:20.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f2101315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:20.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f2101315640 1 -- 172.21.15.190:0/3174938308 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20fc1d3fd0 con 0x7f20fc142a20 2024-03-25T18:25:20.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f2101315640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:20.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f2101315640 10 monclient: _renew_subs 2024-03-25T18:25:20.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f2101315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:20.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f2101315640 1 -- 172.21.15.190:0/3174938308 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20fc1d4f40 con 0x7f20fc142a20 2024-03-25T18:25:20.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.271+0000 7f2101315640 1 librados: init done 2024-03-25T18:25:20.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.273+0000 7f2101315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:20.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.273+0000 7f2101315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:20.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.273+0000 7f2101315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:20.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.273+0000 7f2101315640 1 -- 172.21.15.190:0/3174938308 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20fc0ae3a0 con 0x7f20fc142a20 2024-03-25T18:25:20.277 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.275+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20f4001fe0 con 0x7f20fc142a20 2024-03-25T18:25:20.277 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.275+0000 7f20f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:20.277 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.275+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f20f4003990 con 0x7f20fc142a20 2024-03-25T18:25:20.277 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.275+0000 7f20f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.277 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.275+0000 7f20f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:20.277 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.275+0000 7f20f37fe640 10 monclient: dump: 2024-03-25T18:25:20.277 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.277 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.277 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.277 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.278 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.278 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.278 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.278 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.278 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.278 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.275+0000 7f20f9ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:20.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.275+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f20f4003e80 con 0x7f20fc142a20 2024-03-25T18:25:20.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.275+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1126151693 0 0) 0x7f20f4005570 con 0x7f20fc142a20 2024-03-25T18:25:20.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.275+0000 7f20f37fe640 10 client.4144.objecter ms_dispatch 0x7f20fc000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:25:20.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.275+0000 7f20f37fe640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:25:20.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.275+0000 7f20f37fe640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:25:20.279 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.275+0000 7f20f37fe640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T18:25:20.371 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.370+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f20f4005170 con 0x7f20fc142a20 2024-03-25T18:25:20.371 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.370+0000 7f20f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:20.371 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.370+0000 7f20f37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:20.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.464+0000 7f2101315640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "031772d9-ed09-4828-87a7-6354210e6cc6", "id": 4}] 2024-03-25T18:25:20.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.465+0000 7f2101315640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "031772d9-ed09-4828-87a7-6354210e6cc6", "id": 4}] 2024-03-25T18:25:20.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.465+0000 7f2101315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:20.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.465+0000 7f2101315640 1 -- 172.21.15.190:0/3174938308 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd new", "uuid": "031772d9-ed09-4828-87a7-6354210e6cc6", "id": 4} v 0) v1 -- 0x7f20fc04f730 con 0x7f20fc142a20 2024-03-25T18:25:20.525 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.525+0000 7f20f37fe640 1 -- 172.21.15.190:0/3174938308 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "031772d9-ed09-4828-87a7-6354210e6cc6", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 418480026 0 1697012583) 0x7f20f4005780 con 0x7f20fc142a20 2024-03-25T18:25:20.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.525+0000 7f20f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "031772d9-ed09-4828-87a7-6354210e6cc6", "id": 4}] 2024-03-25T18:25:20.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.525+0000 7f20f37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:20.526 INFO:teuthology.orchestra.run.smithi190.stdout:4 2024-03-25T18:25:20.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.525+0000 7f2101315640 10 librados: watch_flush enter 2024-03-25T18:25:20.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.525+0000 7f2101315640 10 librados: watch_flush exit 2024-03-25T18:25:20.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.525+0000 7f2101315640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T18:25:20.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.525+0000 7f2101315640 10 client.4144.objecter successfully canceled tick 2024-03-25T18:25:20.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.525+0000 7f2101315640 10 monclient: shutdown 2024-03-25T18:25:20.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.525+0000 7f2101315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:20.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.525+0000 7f2101315640 1 -- 172.21.15.190:0/3174938308 >> v1:172.21.15.190:6789/0 conn(0x7f20fc142a20 legacy=0x7f20fc0ac9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.526+0000 7f2101315640 1 -- 172.21.15.190:0/3174938308 shutdown_connections 2024-03-25T18:25:20.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.526+0000 7f2101315640 1 -- 172.21.15.190:0/3174938308 >> 172.21.15.190:0/3174938308 conn(0x7f20fc0668e0 msgr2=0x7f20fc0b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:20.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.526+0000 7f2101315640 1 -- 172.21.15.190:0/3174938308 shutdown_connections 2024-03-25T18:25:20.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.526+0000 7f2101315640 1 -- 172.21.15.190:0/3174938308 wait complete. 2024-03-25T18:25:20.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.526+0000 7f2101315640 1 librados: shutdown 2024-03-25T18:25:20.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 553b866a-dbf1-48fd-8366-8c7522e3e20e 5 2024-03-25T18:25:20.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.636+0000 7fce369cf640 10 monclient: get_monmap_and_config 2024-03-25T18:25:20.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.636+0000 7fce369cf640 10 monclient: build_initial_monmap 2024-03-25T18:25:20.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.636+0000 7fce369cf640 10 monclient: monmap: 2024-03-25T18:25:20.638 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:20.638 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.638 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:25:20.637643+0000 2024-03-25T18:25:20.638 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:25:20.637643+0000 2024-03-25T18:25:20.638 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:20.638 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.638 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:20.638 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:25:20.638 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:25:20.638 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.637+0000 7fce369cf640 1 Processor -- start 2024-03-25T18:25:20.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.637+0000 7fce369cf640 1 -- start start 2024-03-25T18:25:20.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.637+0000 7fce369cf640 10 monclient: init 2024-03-25T18:25:20.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.637+0000 7fce369cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:20.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.637+0000 7fce369cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:20.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.637+0000 7fce369cf640 10 monclient(hunting): picked mon.noname-c con 0x7fce300b4700 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:20.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.637+0000 7fce369cf640 10 monclient(hunting): picked mon.noname-b con 0x7fce30142a20 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:20.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.637+0000 7fce369cf640 10 monclient(hunting): picked mon.noname-a con 0x7fce300a96b0 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:20.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.637+0000 7fce369cf640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce300a9d60 con 0x7fce300a96b0 2024-03-25T18:25:20.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.637+0000 7fce369cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce3013ff10 con 0x7fce300b4700 2024-03-25T18:25:20.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.637+0000 7fce369cf640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce300aa1f0 con 0x7fce30142a20 2024-03-25T18:25:20.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.637+0000 7fce369cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:20.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.637+0000 7fce369cf640 10 monclient(hunting): authenticate will time out at 7475.982643s 2024-03-25T18:25:20.641 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.638+0000 7fce34f6c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fce300a96b0 0x7fce3013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.190:41414/0 (socket says 172.21.15.190:41414) 2024-03-25T18:25:20.641 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.638+0000 7fce34f6c640 1 -- 172.21.15.190:0/2049811005 learned_addr learned my addr 172.21.15.190:0/2049811005 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:20.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 1 -- 172.21.15.190:0/2049811005 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994904206 0 0) 0x7fce300aa1f0 con 0x7fce30142a20 2024-03-25T18:25:20.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:25:20.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 1 -- 172.21.15.190:0/2049811005 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce18003520 con 0x7fce30142a20 2024-03-25T18:25:20.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 1 -- 172.21.15.190:0/2049811005 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891856103 0 0) 0x7fce300a9d60 con 0x7fce300a96b0 2024-03-25T18:25:20.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:25:20.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 1 -- 172.21.15.190:0/2049811005 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce300aa1f0 con 0x7fce300a96b0 2024-03-25T18:25:20.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 1 -- 172.21.15.190:0/2049811005 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582328302 0 0) 0x7fce3013ff10 con 0x7fce300b4700 2024-03-25T18:25:20.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:25:20.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.641+0000 7fce2effd640 1 -- 172.21.15.190:0/2049811005 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce300a9d60 con 0x7fce300b4700 2024-03-25T18:25:20.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 1 -- 172.21.15.190:0/2049811005 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2396047962 0 0) 0x7fce300aa1f0 con 0x7fce300a96b0 2024-03-25T18:25:20.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 1 -- 172.21.15.190:0/2049811005 >> v1:172.21.15.99:6790/0 conn(0x7fce30142a20 legacy=0x7fce3013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 1 -- 172.21.15.190:0/2049811005 >> v1:172.21.15.190:6789/0 conn(0x7fce300b4700 legacy=0x7fce30142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:20.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 1 monclient: found mon.noname-a 2024-03-25T18:25:20.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:20.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:25:20.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 1 -- 172.21.15.190:0/2049811005 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce300b4ae0 con 0x7fce300a96b0 2024-03-25T18:25:20.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 10 monclient: _finish_auth 0 2024-03-25T18:25:20.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:20.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:20.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce369cf640 5 monclient: authenticate success, global_id 4146 2024-03-25T18:25:20.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce369cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:20.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 1 -- 172.21.15.190:0/2049811005 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fce28001930 con 0x7fce300a96b0 2024-03-25T18:25:20.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:20.646 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce2effd640 10 monclient: dump: 2024-03-25T18:25:20.646 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.646 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.646 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.646 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.647 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.647 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.647 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.647 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.647 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.647 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.647 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.644+0000 7fce369cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:20.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.645+0000 7fce2effd640 1 -- 172.21.15.190:0/2049811005 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce28001bc0 con 0x7fce300a96b0 2024-03-25T18:25:20.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.645+0000 7fce2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:20.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.645+0000 7fce369cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:20.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.645+0000 7fce369cf640 4 set_mon_vals no callback set 2024-03-25T18:25:20.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.646+0000 7fce2effd640 1 -- 172.21.15.190:0/2049811005 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fce28001f50 con 0x7fce300a96b0 2024-03-25T18:25:20.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.646+0000 7fce2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.646+0000 7fce2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:20.650 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.646+0000 7fce2effd640 10 monclient: dump: 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.646+0000 7fce369cf640 10 monclient: shutdown 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.646+0000 7fce369cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.646+0000 7fce369cf640 1 -- 172.21.15.190:0/2049811005 >> v1:172.21.15.99:6789/0 conn(0x7fce300a96b0 legacy=0x7fce3013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.646+0000 7fce369cf640 1 -- 172.21.15.190:0/2049811005 shutdown_connections 2024-03-25T18:25:20.651 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.646+0000 7fce369cf640 1 -- 172.21.15.190:0/2049811005 >> 172.21.15.190:0/2049811005 conn(0x7fce300668e0 msgr2=0x7fce30066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.646+0000 7fce369cf640 1 -- 172.21.15.190:0/2049811005 shutdown_connections 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.646+0000 7fce369cf640 1 -- 172.21.15.190:0/2049811005 wait complete. 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.647+0000 7fce369cf640 10 monclient: build_initial_monmap 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.647+0000 7fce369cf640 10 monclient: monmap: 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.647+0000 7fce369cf640 1 librados: starting msgr at 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.647+0000 7fce369cf640 1 librados: starting objecter 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.647+0000 7fce369cf640 1 Processor -- start 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.647+0000 7fce369cf640 1 -- start start 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.647+0000 7fce369cf640 1 librados: setting wanted keys 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.647+0000 7fce369cf640 1 librados: calling monclient init 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.647+0000 7fce369cf640 10 monclient: init 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.648+0000 7fce369cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.648+0000 7fce369cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.648+0000 7fce369cf640 10 monclient(hunting): picked mon.noname-a con 0x7fce300a96b0 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.648+0000 7fce369cf640 10 monclient(hunting): picked mon.noname-b con 0x7fce300b4700 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.648+0000 7fce369cf640 10 monclient(hunting): picked mon.noname-c con 0x7fce30142a20 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.648+0000 7fce369cf640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce301d7680 con 0x7fce300a96b0 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.648+0000 7fce369cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce301d7280 con 0x7fce300b4700 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.648+0000 7fce369cf640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce300ab880 con 0x7fce30142a20 2024-03-25T18:25:20.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.648+0000 7fce369cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:20.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.648+0000 7fce369cf640 10 monclient(hunting): authenticate will time out at 7475.993037s 2024-03-25T18:25:20.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.648+0000 7fce2ffff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fce300a96b0 0x7fce301d7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.190:41420/0 (socket says 172.21.15.190:41420) 2024-03-25T18:25:20.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.648+0000 7fce2ffff640 1 -- 172.21.15.190:0/3302550758 learned_addr learned my addr 172.21.15.190:0/3302550758 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:20.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.649+0000 7fce2cff9640 10 client.?.objecter ms_handle_connect 0x7fce300a96b0 2024-03-25T18:25:20.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.649+0000 7fce2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:20.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.649+0000 7fce2cff9640 10 client.?.objecter ms_handle_connect 0x7fce30142a20 2024-03-25T18:25:20.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.649+0000 7fce2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:20.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.649+0000 7fce2cff9640 10 client.?.objecter ms_handle_connect 0x7fce300b4700 2024-03-25T18:25:20.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.649+0000 7fce2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:20.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184500685 0 0) 0x7fce301d7680 con 0x7fce300a96b0 2024-03-25T18:25:20.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:25:20.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce1c004eb0 con 0x7fce300a96b0 2024-03-25T18:25:20.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151984502 0 0) 0x7fce300ab880 con 0x7fce30142a20 2024-03-25T18:25:20.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:25:20.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce301d7680 con 0x7fce30142a20 2024-03-25T18:25:20.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15923964 0 0) 0x7fce301d7280 con 0x7fce300b4700 2024-03-25T18:25:20.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:25:20.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.651+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce300ab880 con 0x7fce300b4700 2024-03-25T18:25:20.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.655+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2436053965 0 0) 0x7fce301d7680 con 0x7fce30142a20 2024-03-25T18:25:20.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.655+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce301d7280 con 0x7fce30142a20 2024-03-25T18:25:20.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.655+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fce28001930 con 0x7fce30142a20 2024-03-25T18:25:20.657 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.655+0000 7fce2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.655+0000 7fce2cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:20.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.655+0000 7fce2cff9640 10 monclient(hunting): dump: 2024-03-25T18:25:20.658 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.658 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.658 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.658 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.658 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.658 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.658 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.658 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.658 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.658 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.655+0000 7fce2cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:20.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.655+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3647873432 0 0) 0x7fce1c004eb0 con 0x7fce300a96b0 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.655+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce301d7680 con 0x7fce300a96b0 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.655+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fce20001980 con 0x7fce300a96b0 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.655+0000 7fce2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.655+0000 7fce2cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.655+0000 7fce2cff9640 10 monclient(hunting): dump: 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.659 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.657+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 78824473 0 0) 0x7fce300ab880 con 0x7fce300b4700 2024-03-25T18:25:20.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.657+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce1c004eb0 con 0x7fce300b4700 2024-03-25T18:25:20.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.657+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fce24001af0 con 0x7fce300b4700 2024-03-25T18:25:20.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.657+0000 7fce2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.657+0000 7fce2cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:20.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.657+0000 7fce2cff9640 10 monclient(hunting): dump: 2024-03-25T18:25:20.662 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3906721520 0 0) 0x7fce301d7280 con 0x7fce30142a20 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 >> v1:172.21.15.190:6789/0 conn(0x7fce300b4700 legacy=0x7fce30130ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 >> v1:172.21.15.99:6789/0 conn(0x7fce300a96b0 legacy=0x7fce301d7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce2cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce2cff9640 1 monclient: found mon.c 2024-03-25T18:25:20.663 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce2cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce2cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce300aba10 con 0x7fce30142a20 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce2cff9640 10 monclient: _finish_auth 0 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce2cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce369cf640 5 monclient: authenticate success, global_id 4148 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce369cf640 10 monclient: _renew_subs 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce369cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce369cf640 1 -- 172.21.15.190:0/3302550758 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce300ac890 con 0x7fce30142a20 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce369cf640 10 client.4148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce369cf640 10 monclient: _renew_subs 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce369cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce369cf640 1 -- 172.21.15.190:0/3302550758 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce300ad440 con 0x7fce30142a20 2024-03-25T18:25:20.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.659+0000 7fce369cf640 1 librados: init done 2024-03-25T18:25:20.665 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.660+0000 7fce369cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:20.665 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.660+0000 7fce369cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:20.665 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.660+0000 7fce369cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:20.665 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.660+0000 7fce369cf640 1 -- 172.21.15.190:0/3302550758 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3004f730 con 0x7fce30142a20 2024-03-25T18:25:20.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.753+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce28001420 con 0x7fce30142a20 2024-03-25T18:25:20.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.753+0000 7fce2cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:20.755 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.753+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fce28004fb0 con 0x7fce30142a20 2024-03-25T18:25:20.755 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.753+0000 7fce2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.755 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.753+0000 7fce2cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:20.755 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.753+0000 7fce2cff9640 10 monclient: dump: 2024-03-25T18:25:20.755 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.755 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.755 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.755 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.755 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.755 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.755 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.755 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.755 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.755 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.753+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fce280054a0 con 0x7fce30142a20 2024-03-25T18:25:20.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.753+0000 7fce2e7fc640 4 set_mon_vals no callback set 2024-03-25T18:25:20.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.754+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2628329029 0 0) 0x7fce28006a40 con 0x7fce30142a20 2024-03-25T18:25:20.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.754+0000 7fce2cff9640 10 client.4148.objecter ms_dispatch 0x7fce30000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:25:20.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.754+0000 7fce2cff9640 3 client.4148.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:25:20.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.754+0000 7fce2cff9640 3 client.4148.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:25:20.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.754+0000 7fce2cff9640 20 client.4148.objecter dump_active .. 0 homeless 2024-03-25T18:25:20.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.754+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fce280064c0 con 0x7fce30142a20 2024-03-25T18:25:20.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.754+0000 7fce2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:20.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.754+0000 7fce2cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:20.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.827+0000 7fce369cf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "553b866a-dbf1-48fd-8366-8c7522e3e20e", "id": 5}] 2024-03-25T18:25:20.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.827+0000 7fce369cf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "553b866a-dbf1-48fd-8366-8c7522e3e20e", "id": 5}] 2024-03-25T18:25:20.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.827+0000 7fce369cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:20.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.827+0000 7fce369cf640 1 -- 172.21.15.190:0/3302550758 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd new", "uuid": "553b866a-dbf1-48fd-8366-8c7522e3e20e", "id": 5} v 0) v1 -- 0x7fce301d7410 con 0x7fce30142a20 2024-03-25T18:25:20.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.857+0000 7fce2cff9640 1 -- 172.21.15.190:0/3302550758 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "553b866a-dbf1-48fd-8366-8c7522e3e20e", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2442626023 0 1988412176) 0x7fce280067b0 con 0x7fce30142a20 2024-03-25T18:25:20.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.857+0000 7fce2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "553b866a-dbf1-48fd-8366-8c7522e3e20e", "id": 5}] 2024-03-25T18:25:20.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.857+0000 7fce2cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:20.857 INFO:teuthology.orchestra.run.smithi190.stdout:5 2024-03-25T18:25:20.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.857+0000 7fce369cf640 10 librados: watch_flush enter 2024-03-25T18:25:20.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.857+0000 7fce369cf640 10 librados: watch_flush exit 2024-03-25T18:25:20.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.857+0000 7fce369cf640 20 client.4148.objecter shutdown clearing up homeless session... 2024-03-25T18:25:20.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.857+0000 7fce369cf640 10 client.4148.objecter successfully canceled tick 2024-03-25T18:25:20.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.858+0000 7fce369cf640 10 monclient: shutdown 2024-03-25T18:25:20.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.858+0000 7fce369cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:20.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.858+0000 7fce369cf640 1 -- 172.21.15.190:0/3302550758 >> v1:172.21.15.99:6790/0 conn(0x7fce30142a20 legacy=0x7fce301e2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.858+0000 7fce369cf640 1 -- 172.21.15.190:0/3302550758 shutdown_connections 2024-03-25T18:25:20.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.858+0000 7fce369cf640 1 -- 172.21.15.190:0/3302550758 >> 172.21.15.190:0/3302550758 conn(0x7fce300668e0 msgr2=0x7fce30066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:20.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.858+0000 7fce369cf640 1 -- 172.21.15.190:0/3302550758 shutdown_connections 2024-03-25T18:25:20.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.858+0000 7fce369cf640 1 -- 172.21.15.190:0/3302550758 wait complete. 2024-03-25T18:25:20.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.858+0000 7fce369cf640 1 librados: shutdown 2024-03-25T18:25:20.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new da817edd-3a2f-4c44-b911-5d3fb2ebb686 6 2024-03-25T18:25:20.914 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:20.914+0000 7f761b873040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:25:20.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.969+0000 7f8fd5759640 10 monclient: get_monmap_and_config 2024-03-25T18:25:20.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.969+0000 7f8fd5759640 10 monclient: build_initial_monmap 2024-03-25T18:25:20.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.969+0000 7f8fd5759640 10 monclient: monmap: 2024-03-25T18:25:20.969 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:20.969 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.969 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:25:20.969255+0000 2024-03-25T18:25:20.969 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:25:20.969255+0000 2024-03-25T18:25:20.969 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:20.969 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.969 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:20.969 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:25:20.970 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:25:20.970 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.969+0000 7f8fd5759640 1 Processor -- start 2024-03-25T18:25:20.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.969+0000 7f8fd5759640 1 -- start start 2024-03-25T18:25:20.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.969+0000 7f8fd5759640 10 monclient: init 2024-03-25T18:25:20.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.970+0000 7f8fd5759640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:20.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.970+0000 7f8fd5759640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:20.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.970+0000 7f8fd5759640 10 monclient(hunting): picked mon.noname-c con 0x7f8fd00b4700 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:20.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.970+0000 7f8fd5759640 10 monclient(hunting): picked mon.noname-b con 0x7f8fd0142a20 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:20.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.970+0000 7f8fd5759640 10 monclient(hunting): picked mon.noname-a con 0x7f8fd00a96b0 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:20.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.970+0000 7f8fd5759640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd00a9d60 con 0x7f8fd00a96b0 2024-03-25T18:25:20.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.970+0000 7f8fd5759640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd013ff10 con 0x7f8fd00b4700 2024-03-25T18:25:20.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.970+0000 7f8fd5759640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd00aa1f0 con 0x7f8fd0142a20 2024-03-25T18:25:20.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.970+0000 7f8fd5759640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:20.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.970+0000 7f8fd5759640 10 monclient(hunting): authenticate will time out at 7476.314198s 2024-03-25T18:25:20.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.971+0000 7f8fcf7fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f8fd00a96b0 0x7f8fd013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.190:41434/0 (socket says 172.21.15.190:41434) 2024-03-25T18:25:20.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.971+0000 7f8fcf7fe640 1 -- 172.21.15.190:0/2848741340 learned_addr learned my addr 172.21.15.190:0/2848741340 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:20.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 1 -- 172.21.15.190:0/2848741340 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826408920 0 0) 0x7f8fd013ff10 con 0x7f8fd00b4700 2024-03-25T18:25:20.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:25:20.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 1 -- 172.21.15.190:0/2848741340 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fb8003520 con 0x7f8fd00b4700 2024-03-25T18:25:20.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 1 -- 172.21.15.190:0/2848741340 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820292951 0 0) 0x7f8fd00aa1f0 con 0x7f8fd0142a20 2024-03-25T18:25:20.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:25:20.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 1 -- 172.21.15.190:0/2848741340 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd013ff10 con 0x7f8fd0142a20 2024-03-25T18:25:20.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 1 -- 172.21.15.190:0/2848741340 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018565922 0 0) 0x7f8fd00a9d60 con 0x7f8fd00a96b0 2024-03-25T18:25:20.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.975 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.975 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.975 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:25:20.975 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.973+0000 7f8fcdffb640 1 -- 172.21.15.190:0/2848741340 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd00aa1f0 con 0x7f8fd00a96b0 2024-03-25T18:25:20.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 1 -- 172.21.15.190:0/2848741340 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4037802981 0 0) 0x7f8fb8003520 con 0x7f8fd00b4700 2024-03-25T18:25:20.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 1 -- 172.21.15.190:0/2848741340 >> v1:172.21.15.99:6790/0 conn(0x7f8fd0142a20 legacy=0x7f8fd013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 1 -- 172.21.15.190:0/2848741340 >> v1:172.21.15.99:6789/0 conn(0x7f8fd00a96b0 legacy=0x7f8fd013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:20.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 1 monclient: found mon.noname-c 2024-03-25T18:25:20.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:20.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:25:20.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 1 -- 172.21.15.190:0/2848741340 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd00b4ae0 con 0x7f8fd00b4700 2024-03-25T18:25:20.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:20.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:20.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:20.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 1 -- 172.21.15.190:0/2848741340 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8fc4001690 con 0x7f8fd00b4700 2024-03-25T18:25:20.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fd5759640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fd5759640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fcdffb640 10 monclient: dump: 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.977 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.976+0000 7f8fd5759640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:20.978 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fcdffb640 1 -- 172.21.15.190:0/2848741340 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fc40017f0 con 0x7f8fd00b4700 2024-03-25T18:25:20.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fcdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:20.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fcdffb640 1 -- 172.21.15.190:0/2848741340 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8fc4001c00 con 0x7f8fd00b4700 2024-03-25T18:25:20.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fd5759640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:20.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fd5759640 4 set_mon_vals no callback set 2024-03-25T18:25:20.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fcdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:20.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fcdffb640 10 monclient: dump: 2024-03-25T18:25:20.979 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.979 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.979 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.979 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.979 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.979 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.980 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.980 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.980 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.980 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fd5759640 10 monclient: shutdown 2024-03-25T18:25:20.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fd5759640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:20.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fd5759640 1 -- 172.21.15.190:0/2848741340 >> v1:172.21.15.190:6789/0 conn(0x7f8fd00b4700 legacy=0x7f8fd0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fd5759640 1 -- 172.21.15.190:0/2848741340 shutdown_connections 2024-03-25T18:25:20.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fd5759640 1 -- 172.21.15.190:0/2848741340 >> 172.21.15.190:0/2848741340 conn(0x7f8fd00668e0 msgr2=0x7f8fd0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:20.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fd5759640 1 -- 172.21.15.190:0/2848741340 shutdown_connections 2024-03-25T18:25:20.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.978+0000 7f8fd5759640 1 -- 172.21.15.190:0/2848741340 wait complete. 2024-03-25T18:25:20.982 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.979+0000 7f8fd5759640 10 monclient: build_initial_monmap 2024-03-25T18:25:20.982 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.979+0000 7f8fd5759640 10 monclient: monmap: 2024-03-25T18:25:20.982 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:20.982 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:20.982 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.979+0000 7f8fd5759640 1 librados: starting msgr at 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.979+0000 7f8fd5759640 1 librados: starting objecter 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 1 Processor -- start 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 1 -- start start 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 1 librados: setting wanted keys 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 1 librados: calling monclient init 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 10 monclient: init 2024-03-25T18:25:20.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 10 monclient(hunting): picked mon.noname-b con 0x7f8fd00a96b0 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 10 monclient(hunting): picked mon.noname-a con 0x7f8fd00b4700 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 10 monclient(hunting): picked mon.noname-c con 0x7f8fd0142a20 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd00aed20 con 0x7f8fd00b4700 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd00af000 con 0x7f8fd00a96b0 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd00ad2b0 con 0x7f8fd0142a20 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.980+0000 7f8fd5759640 10 monclient(hunting): authenticate will time out at 7476.324548s 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.981+0000 7f8fce7fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f8fd00b4700 0x7f8fd01df060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.190:41436/0 (socket says 172.21.15.190:41436) 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.981+0000 7f8fce7fc640 1 -- 172.21.15.190:0/1084387923 learned_addr learned my addr 172.21.15.190:0/1084387923 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.981+0000 7f8fb77fe640 10 client.?.objecter ms_handle_connect 0x7f8fd00b4700 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.981+0000 7f8fb77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.981+0000 7f8fb77fe640 10 client.?.objecter ms_handle_connect 0x7f8fd00a96b0 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.981+0000 7f8fb77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:20.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.981+0000 7f8fb77fe640 10 client.?.objecter ms_handle_connect 0x7f8fd0142a20 2024-03-25T18:25:20.985 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.981+0000 7f8fb77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:20.985 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792501040 0 0) 0x7f8fd00af000 con 0x7f8fd00a96b0 2024-03-25T18:25:20.985 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:25:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fbc004eb0 con 0x7f8fd00a96b0 2024-03-25T18:25:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063124679 0 0) 0x7f8fd00ad2b0 con 0x7f8fd0142a20 2024-03-25T18:25:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:25:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd00af000 con 0x7f8fd0142a20 2024-03-25T18:25:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735587074 0 0) 0x7f8fd00aed20 con 0x7f8fd00b4700 2024-03-25T18:25:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:20.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:25:20.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.983+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd00ad2b0 con 0x7f8fd00b4700 2024-03-25T18:25:20.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.988+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2338506042 0 0) 0x7f8fd00af000 con 0x7f8fd0142a20 2024-03-25T18:25:20.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.988+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fd00aed20 con 0x7f8fd0142a20 2024-03-25T18:25:20.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.988+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8fc8001bf0 con 0x7f8fd0142a20 2024-03-25T18:25:20.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.988+0000 7f8fb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.988+0000 7f8fb77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:20.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.988+0000 7f8fb77fe640 10 monclient(hunting): dump: 2024-03-25T18:25:20.988 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.988+0000 7f8fb77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.988+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2752975536 0 0) 0x7f8fbc004eb0 con 0x7f8fd00a96b0 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.988+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fd00af000 con 0x7f8fd00a96b0 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.988+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8fc4001b00 con 0x7f8fd00a96b0 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.988+0000 7f8fb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.988+0000 7f8fb77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:20.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.988+0000 7f8fb77fe640 10 monclient(hunting): dump: 2024-03-25T18:25:20.990 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.990 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.990 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.990 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.990 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.990 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.990 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.990 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.990 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.990 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.989+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3834148907 0 0) 0x7f8fd00ad2b0 con 0x7f8fd00b4700 2024-03-25T18:25:20.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.989+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fbc004eb0 con 0x7f8fd00b4700 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.989+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8fc00018d0 con 0x7f8fd00b4700 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.989+0000 7f8fb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.989+0000 7f8fb77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.989+0000 7f8fb77fe640 10 monclient(hunting): dump: 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:20.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3698207224 0 0) 0x7f8fd00aed20 con 0x7f8fd0142a20 2024-03-25T18:25:20.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 >> v1:172.21.15.190:6789/0 conn(0x7f8fd00a96b0 legacy=0x7f8fd00af420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 >> v1:172.21.15.99:6789/0 conn(0x7f8fd00b4700 legacy=0x7f8fd01df060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:20.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fb77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:20.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fb77fe640 1 monclient: found mon.c 2024-03-25T18:25:20.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:20.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fb77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:20.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd00afb30 con 0x7f8fd0142a20 2024-03-25T18:25:20.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fb77fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:20.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fb77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:20.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:20.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fd5759640 5 monclient: authenticate success, global_id 4154 2024-03-25T18:25:20.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fd5759640 10 monclient: _renew_subs 2024-03-25T18:25:20.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fd5759640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:20.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fd5759640 1 -- 172.21.15.190:0/1084387923 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd00ac9a0 con 0x7f8fd0142a20 2024-03-25T18:25:20.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fd5759640 10 client.4154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:20.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fd5759640 10 monclient: _renew_subs 2024-03-25T18:25:20.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fd5759640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:20.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fd5759640 1 -- 172.21.15.190:0/1084387923 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd01358e0 con 0x7f8fd0142a20 2024-03-25T18:25:20.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.991+0000 7f8fd5759640 1 librados: init done 2024-03-25T18:25:20.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.992+0000 7f8fd5759640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:20.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.992+0000 7f8fd5759640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:20.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.992+0000 7f8fd5759640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:20.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:20.992+0000 7f8fd5759640 1 -- 172.21.15.190:0/1084387923 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd00ad440 con 0x7f8fd0142a20 2024-03-25T18:25:21.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.090+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fc8002000 con 0x7f8fd0142a20 2024-03-25T18:25:21.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.090+0000 7f8fb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:21.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.090+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8fc8004d00 con 0x7f8fd0142a20 2024-03-25T18:25:21.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.090+0000 7f8fb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:21.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.090+0000 7f8fb77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:21.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.090+0000 7f8fb77fe640 10 monclient: dump: 2024-03-25T18:25:21.091 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:21.091 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:21.091 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:21.091 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:21.091 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:21.091 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:21.091 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:21.091 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:21.091 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:21.091 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:21.091 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.090+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8fc80052b0 con 0x7f8fd0142a20 2024-03-25T18:25:21.091 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.090+0000 7f8fcd7fa640 4 set_mon_vals no callback set 2024-03-25T18:25:21.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.092+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2960033429 0 0) 0x7f8fc8006a10 con 0x7f8fd0142a20 2024-03-25T18:25:21.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.092+0000 7f8fb77fe640 10 client.4154.objecter ms_dispatch 0x7f8fd0000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:25:21.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.092+0000 7f8fb77fe640 3 client.4154.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:25:21.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.092+0000 7f8fb77fe640 3 client.4154.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:25:21.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.092+0000 7f8fb77fe640 20 client.4154.objecter dump_active .. 0 homeless 2024-03-25T18:25:21.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.092+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8fc8006420 con 0x7f8fd0142a20 2024-03-25T18:25:21.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.092+0000 7f8fb77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:21.092 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.092+0000 7f8fb77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:21.104 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:21.103+0000 7ff11184b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:25:21.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.164+0000 7f8fd5759640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "da817edd-3a2f-4c44-b911-5d3fb2ebb686", "id": 6}] 2024-03-25T18:25:21.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.164+0000 7f8fd5759640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "da817edd-3a2f-4c44-b911-5d3fb2ebb686", "id": 6}] 2024-03-25T18:25:21.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.164+0000 7f8fd5759640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:21.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.164+0000 7f8fd5759640 1 -- 172.21.15.190:0/1084387923 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd new", "uuid": "da817edd-3a2f-4c44-b911-5d3fb2ebb686", "id": 6} v 0) v1 -- 0x7f8fd004f730 con 0x7f8fd0142a20 2024-03-25T18:25:21.195 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:21.195+0000 7f761b873040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:25:21.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.212+0000 7f8fb77fe640 1 -- 172.21.15.190:0/1084387923 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "da817edd-3a2f-4c44-b911-5d3fb2ebb686", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2797980045 0 1113811849) 0x7f8fc8006710 con 0x7f8fd0142a20 2024-03-25T18:25:21.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.212+0000 7f8fb77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "da817edd-3a2f-4c44-b911-5d3fb2ebb686", "id": 6}] 2024-03-25T18:25:21.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.212+0000 7f8fb77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:21.212 INFO:teuthology.orchestra.run.smithi190.stdout:6 2024-03-25T18:25:21.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.213+0000 7f8fd5759640 10 librados: watch_flush enter 2024-03-25T18:25:21.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.213+0000 7f8fd5759640 10 librados: watch_flush exit 2024-03-25T18:25:21.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.213+0000 7f8fd5759640 20 client.4154.objecter shutdown clearing up homeless session... 2024-03-25T18:25:21.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.213+0000 7f8fd5759640 10 client.4154.objecter successfully canceled tick 2024-03-25T18:25:21.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.213+0000 7f8fd5759640 10 monclient: shutdown 2024-03-25T18:25:21.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.213+0000 7f8fd5759640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:21.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.213+0000 7f8fd5759640 1 -- 172.21.15.190:0/1084387923 >> v1:172.21.15.99:6790/0 conn(0x7f8fd0142a20 legacy=0x7f8fd01e2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:21.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.213+0000 7f8fd5759640 1 -- 172.21.15.190:0/1084387923 shutdown_connections 2024-03-25T18:25:21.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.213+0000 7f8fd5759640 1 -- 172.21.15.190:0/1084387923 >> 172.21.15.190:0/1084387923 conn(0x7f8fd00668e0 msgr2=0x7f8fd0066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:21.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.213+0000 7f8fd5759640 1 -- 172.21.15.190:0/1084387923 shutdown_connections 2024-03-25T18:25:21.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.213+0000 7f8fd5759640 1 -- 172.21.15.190:0/1084387923 wait complete. 2024-03-25T18:25:21.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.213+0000 7f8fd5759640 1 librados: shutdown 2024-03-25T18:25:21.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new a443b4cf-d3f4-4411-8f9c-ee6f635d2504 7 2024-03-25T18:25:21.251 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:21.252+0000 7f761b873040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:25:21.323 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.323+0000 7fc5adfe6640 10 monclient: get_monmap_and_config 2024-03-25T18:25:21.323 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.323+0000 7fc5adfe6640 10 monclient: build_initial_monmap 2024-03-25T18:25:21.323 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.323+0000 7fc5adfe6640 10 monclient: monmap: 2024-03-25T18:25:21.323 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:21.323 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:21.323 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:25:21.323318+0000 2024-03-25T18:25:21.323 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:25:21.323318+0000 2024-03-25T18:25:21.323 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:21.324 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:21.324 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:21.324 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:25:21.324 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:25:21.324 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:21.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.324+0000 7fc5adfe6640 1 Processor -- start 2024-03-25T18:25:21.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.324+0000 7fc5adfe6640 1 -- start start 2024-03-25T18:25:21.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.324+0000 7fc5adfe6640 10 monclient: init 2024-03-25T18:25:21.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.324+0000 7fc5adfe6640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:21.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.324+0000 7fc5adfe6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:21.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.324+0000 7fc5adfe6640 10 monclient(hunting): picked mon.noname-a con 0x7fc5a80b4700 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:21.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.324+0000 7fc5adfe6640 10 monclient(hunting): picked mon.noname-b con 0x7fc5a8142a20 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:21.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.324+0000 7fc5adfe6640 10 monclient(hunting): picked mon.noname-c con 0x7fc5a80a96b0 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:21.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.324+0000 7fc5adfe6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a80a9d60 con 0x7fc5a80b4700 2024-03-25T18:25:21.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.324+0000 7fc5adfe6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a813ff10 con 0x7fc5a80a96b0 2024-03-25T18:25:21.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.324+0000 7fc5adfe6640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a80aa1f0 con 0x7fc5a8142a20 2024-03-25T18:25:21.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.324+0000 7fc5adfe6640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:21.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.324+0000 7fc5adfe6640 10 monclient(hunting): authenticate will time out at 7476.668495s 2024-03-25T18:25:21.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.325+0000 7fc5a77fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc5a80b4700 0x7fc5a8142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.190:41446/0 (socket says 172.21.15.190:41446) 2024-03-25T18:25:21.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.325+0000 7fc5a77fe640 1 -- 172.21.15.190:0/1824460621 learned_addr learned my addr 172.21.15.190:0/1824460621 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:21.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.327+0000 7fc5a6ffd640 1 -- 172.21.15.190:0/1824460621 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690935284 0 0) 0x7fc5a813ff10 con 0x7fc5a80a96b0 2024-03-25T18:25:21.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.327+0000 7fc5a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:21.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.327+0000 7fc5a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:21.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.327+0000 7fc5a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:21.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.327+0000 7fc5a6ffd640 10 monclient(hunting): my global_id is 4159 2024-03-25T18:25:21.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.327+0000 7fc5a6ffd640 1 -- 172.21.15.190:0/1824460621 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc594003520 con 0x7fc5a80a96b0 2024-03-25T18:25:21.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.327+0000 7fc5a6ffd640 1 -- 172.21.15.190:0/1824460621 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689593048 0 0) 0x7fc5a80a9d60 con 0x7fc5a80b4700 2024-03-25T18:25:21.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.327+0000 7fc5a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:21.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.327+0000 7fc5a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:21.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.327+0000 7fc5a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:21.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.327+0000 7fc5a6ffd640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:25:21.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.327+0000 7fc5a6ffd640 1 -- 172.21.15.190:0/1824460621 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a813ff10 con 0x7fc5a80b4700 2024-03-25T18:25:21.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.328+0000 7fc5a6ffd640 1 -- 172.21.15.190:0/1824460621 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845939228 0 0) 0x7fc5a80aa1f0 con 0x7fc5a8142a20 2024-03-25T18:25:21.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.328+0000 7fc5a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:21.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.328+0000 7fc5a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:21.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.328+0000 7fc5a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:21.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.328+0000 7fc5a6ffd640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:25:21.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.328+0000 7fc5a6ffd640 1 -- 172.21.15.190:0/1824460621 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a80a9d60 con 0x7fc5a8142a20 2024-03-25T18:25:21.330 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.330+0000 7fc5a6ffd640 1 -- 172.21.15.190:0/1824460621 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2485588357 0 0) 0x7fc594003520 con 0x7fc5a80a96b0 2024-03-25T18:25:21.330 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.330+0000 7fc5a6ffd640 1 -- 172.21.15.190:0/1824460621 >> v1:172.21.15.99:6790/0 conn(0x7fc5a8142a20 legacy=0x7fc5a813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:21.330 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.330+0000 7fc5a6ffd640 1 -- 172.21.15.190:0/1824460621 >> v1:172.21.15.99:6789/0 conn(0x7fc5a80b4700 legacy=0x7fc5a8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:21.330 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.330+0000 7fc5a6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:21.330 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.330+0000 7fc5a6ffd640 1 monclient: found mon.noname-c 2024-03-25T18:25:21.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.330+0000 7fc5a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:21.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.330+0000 7fc5a6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:25:21.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.330+0000 7fc5a6ffd640 1 -- 172.21.15.190:0/1824460621 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a80b4ae0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.330+0000 7fc5a6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:21.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.330+0000 7fc5a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:21.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.330+0000 7fc5a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:21.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.330+0000 7fc5adfe6640 5 monclient: authenticate success, global_id 4159 2024-03-25T18:25:21.331 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.330+0000 7fc5adfe6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:21.332 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.331+0000 7fc5a6ffd640 1 -- 172.21.15.190:0/1824460621 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fc5a00019d0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.332 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.331+0000 7fc5a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:21.332 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.331+0000 7fc5a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:21.332 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.331+0000 7fc5a6ffd640 10 monclient: dump: 2024-03-25T18:25:21.332 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:21.332 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:21.332 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:21.332 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:21.332 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.331+0000 7fc5adfe6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.332+0000 7fc5a6ffd640 1 -- 172.21.15.190:0/1824460621 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5a0001c20 con 0x7fc5a80a96b0 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.332+0000 7fc5a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.332+0000 7fc5a6ffd640 1 -- 172.21.15.190:0/1824460621 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fc5a0001fb0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.332+0000 7fc5a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.332+0000 7fc5a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.332+0000 7fc5a6ffd640 10 monclient: dump: 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:21.334 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:21.334 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:21.334 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:21.334 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:21.334 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:21.334 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:21.334 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:21.334 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.332+0000 7fc5adfe6640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:21.334 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.332+0000 7fc5adfe6640 4 set_mon_vals no callback set 2024-03-25T18:25:21.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.332+0000 7fc5adfe6640 10 monclient: shutdown 2024-03-25T18:25:21.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.332+0000 7fc5adfe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:21.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.332+0000 7fc5adfe6640 1 -- 172.21.15.190:0/1824460621 >> v1:172.21.15.190:6789/0 conn(0x7fc5a80a96b0 legacy=0x7fc5a813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:21.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.333+0000 7fc5adfe6640 1 -- 172.21.15.190:0/1824460621 shutdown_connections 2024-03-25T18:25:21.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.333+0000 7fc5adfe6640 1 -- 172.21.15.190:0/1824460621 >> 172.21.15.190:0/1824460621 conn(0x7fc5a80668e0 msgr2=0x7fc5a8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:21.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.333+0000 7fc5adfe6640 1 -- 172.21.15.190:0/1824460621 shutdown_connections 2024-03-25T18:25:21.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.333+0000 7fc5adfe6640 1 -- 172.21.15.190:0/1824460621 wait complete. 2024-03-25T18:25:21.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.333+0000 7fc5adfe6640 10 monclient: build_initial_monmap 2024-03-25T18:25:21.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.333+0000 7fc5adfe6640 10 monclient: monmap: 2024-03-25T18:25:21.338 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-03-25T18:25:21.338 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:21.338 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-03-25T18:25:21.338 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-03-25T18:25:21.338 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:21.338 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-03-25T18:25:21.338 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:21.338 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:21.338 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.333+0000 7fc5adfe6640 1 librados: starting msgr at 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.333+0000 7fc5adfe6640 1 librados: starting objecter 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 1 Processor -- start 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 1 -- start start 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 1 librados: setting wanted keys 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 1 librados: calling monclient init 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 10 monclient: init 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 10 monclient(hunting): picked mon.noname-b con 0x7fc5a80a96b0 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 10 monclient(hunting): picked mon.noname-a con 0x7fc5a80b4700 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 10 monclient(hunting): picked mon.noname-c con 0x7fc5a8142a20 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a813bcc0 con 0x7fc5a80b4700 2024-03-25T18:25:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a80acc70 con 0x7fc5a80a96b0 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a80adcf0 con 0x7fc5a8142a20 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.334+0000 7fc5adfe6640 10 monclient(hunting): authenticate will time out at 7476.678423s 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.335+0000 7fc59ffff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc5a80b4700 0x7fc5a80ad5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.190:41450/0 (socket says 172.21.15.190:41450) 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.335+0000 7fc59ffff640 1 -- 172.21.15.190:0/2758858181 learned_addr learned my addr 172.21.15.190:0/2758858181 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.335+0000 7fc5a4ff9640 10 client.?.objecter ms_handle_connect 0x7fc5a80a96b0 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.335+0000 7fc5a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.335+0000 7fc5a4ff9640 10 client.?.objecter ms_handle_connect 0x7fc5a80b4700 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.335+0000 7fc5a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.335+0000 7fc5a4ff9640 10 client.?.objecter ms_handle_connect 0x7fc5a8142a20 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.335+0000 7fc5a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637760379 0 0) 0x7fc5a80acc70 con 0x7fc5a80a96b0 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:21.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:21.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 10 monclient(hunting): my global_id is 4162 2024-03-25T18:25:21.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc588004eb0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621883530 0 0) 0x7fc5a813bcc0 con 0x7fc5a80b4700 2024-03-25T18:25:21.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:21.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:21.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:21.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:25:21.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a80acc70 con 0x7fc5a80b4700 2024-03-25T18:25:21.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204314140 0 0) 0x7fc5a80adcf0 con 0x7fc5a8142a20 2024-03-25T18:25:21.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:21.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:21.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:21.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:25:21.341 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.337+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a813bcc0 con 0x7fc5a8142a20 2024-03-25T18:25:21.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.341+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 443944131 0 0) 0x7fc588004eb0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.341+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5a80adcf0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.341+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fc5980018b0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.341+0000 7fc5a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:21.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.341+0000 7fc5a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:21.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.341+0000 7fc5a4ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:21.343 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:21.343 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:21.343 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:21.343 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:21.343 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:21.343 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.341+0000 7fc5a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.342+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1689214925 0 0) 0x7fc5a80acc70 con 0x7fc5a80b4700 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.342+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc588004eb0 con 0x7fc5a80b4700 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.342+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fc5900019f0 con 0x7fc5a80b4700 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.342+0000 7fc5a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.342+0000 7fc5a4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.342+0000 7fc5a4ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:21.344 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.343+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 868945135 0 0) 0x7fc5a813bcc0 con 0x7fc5a8142a20 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.343+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5a80acc70 con 0x7fc5a8142a20 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.343+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fc5a0001d90 con 0x7fc5a8142a20 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.343+0000 7fc5a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.343+0000 7fc5a4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.343+0000 7fc5a4ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:21.345 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:21.346 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:21.346 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:21.346 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:21.346 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:21.346 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:21.346 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:21.347 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3129307893 0 0) 0x7fc5a80adcf0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.347 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 >> v1:172.21.15.99:6790/0 conn(0x7fc5a8142a20 legacy=0x7fc5a81e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:21.347 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 >> v1:172.21.15.99:6789/0 conn(0x7fc5a80b4700 legacy=0x7fc5a80ad5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:21.347 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5a4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:25:21.347 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5a4ff9640 1 monclient: found mon.b 2024-03-25T18:25:21.347 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:21.347 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5a4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:21.347 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a80acee0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.347 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5a4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5a4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5adfe6640 5 monclient: authenticate success, global_id 4162 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5adfe6640 10 monclient: _renew_subs 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5adfe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5adfe6640 1 -- 172.21.15.190:0/2758858181 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a80ad1c0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5adfe6640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5adfe6640 10 monclient: _renew_subs 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5adfe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5adfe6640 1 -- 172.21.15.190:0/2758858181 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a8139ab0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.345+0000 7fc5adfe6640 1 librados: init done 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.346+0000 7fc5adfe6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.346+0000 7fc5adfe6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.346+0000 7fc5adfe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:21.348 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.346+0000 7fc5adfe6640 1 -- 172.21.15.190:0/2758858181 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a813ba90 con 0x7fc5a80a96b0 2024-03-25T18:25:21.372 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:21.373+0000 7f761b873040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:25:21.395 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:21.395+0000 7ff11184b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:25:21.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5980047e0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:21.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fc598004af0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:21.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:21.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 10 monclient: dump: 2024-03-25T18:25:21.442 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-03-25T18:25:21.442 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:21.442 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:21.442 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc598004fe0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a67fc640 4 set_mon_vals no callback set 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1181641952 0 0) 0x7fc5980067e0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 10 client.4162.objecter ms_dispatch 0x7fc5a8000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc598006b00 con 0x7fc5a80a96b0 2024-03-25T18:25:21.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:21.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.441+0000 7fc5a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:21.452 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:21.452+0000 7ff11184b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:25:21.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.513+0000 7fc5adfe6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a443b4cf-d3f4-4411-8f9c-ee6f635d2504", "id": 7}] 2024-03-25T18:25:21.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.513+0000 7fc5adfe6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a443b4cf-d3f4-4411-8f9c-ee6f635d2504", "id": 7}] 2024-03-25T18:25:21.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.513+0000 7fc5adfe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:25:21.514 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.513+0000 7fc5adfe6640 1 -- 172.21.15.190:0/2758858181 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a443b4cf-d3f4-4411-8f9c-ee6f635d2504", "id": 7} v 0) v1 -- 0x7fc5a804f730 con 0x7fc5a80a96b0 2024-03-25T18:25:21.578 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-25T18:25:21.578+0000 7ff11184b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:25:21.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.652+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1213431838 0 0) 0x7fc598006140 con 0x7fc5a80a96b0 2024-03-25T18:25:21.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.907+0000 7fc5a4ff9640 1 -- 172.21.15.190:0/2758858181 <== mon.1 v1:172.21.15.190:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a443b4cf-d3f4-4411-8f9c-ee6f635d2504", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3949343000 0 1371665406) 0x7fc598015bb0 con 0x7fc5a80a96b0 2024-03-25T18:25:21.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.907+0000 7fc5a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a443b4cf-d3f4-4411-8f9c-ee6f635d2504", "id": 7}] 2024-03-25T18:25:21.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.907+0000 7fc5a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:21.908 INFO:teuthology.orchestra.run.smithi190.stdout:7 2024-03-25T18:25:21.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.908+0000 7fc5adfe6640 10 librados: watch_flush enter 2024-03-25T18:25:21.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.908+0000 7fc5adfe6640 10 librados: watch_flush exit 2024-03-25T18:25:21.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.908+0000 7fc5adfe6640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-25T18:25:21.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.908+0000 7fc5adfe6640 10 client.4162.objecter successfully canceled tick 2024-03-25T18:25:21.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.908+0000 7fc5adfe6640 10 monclient: shutdown 2024-03-25T18:25:21.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.908+0000 7fc5adfe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:21.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.908+0000 7fc5adfe6640 1 -- 172.21.15.190:0/2758858181 >> v1:172.21.15.190:6789/0 conn(0x7fc5a80a96b0 legacy=0x7fc5a80abaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:21.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.908+0000 7fc5adfe6640 1 -- 172.21.15.190:0/2758858181 shutdown_connections 2024-03-25T18:25:21.910 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.908+0000 7fc5adfe6640 1 -- 172.21.15.190:0/2758858181 >> 172.21.15.190:0/2758858181 conn(0x7fc5a80668e0 msgr2=0x7fc5a81364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:21.910 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.908+0000 7fc5adfe6640 1 -- 172.21.15.190:0/2758858181 shutdown_connections 2024-03-25T18:25:21.910 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.908+0000 7fc5adfe6640 1 -- 172.21.15.190:0/2758858181 wait complete. 2024-03-25T18:25:21.910 INFO:teuthology.orchestra.run.smithi190.stderr:2024-03-25T18:25:21.909+0000 7fc5adfe6640 1 librados: shutdown 2024-03-25T18:25:21.919 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:21.919 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:25:21.919 DEBUG:teuthology.orchestra.run.smithi099:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:25:21.921 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:25:21.921 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:21.922 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:25:21.922 DEBUG:teuthology.orchestra.run.smithi099:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:25:21.924 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:25:21.924 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:21.924 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:25:21.924 DEBUG:teuthology.orchestra.run.smithi099:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:25:21.926 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:25:21.926 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:21.926 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:25:21.927 DEBUG:teuthology.orchestra.run.smithi099:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:25:21.929 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:25:21.929 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:21.929 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:25:21.929 DEBUG:teuthology.orchestra.run.smithi190:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:25:21.962 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:25:21.962 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:21.962 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:25:21.962 DEBUG:teuthology.orchestra.run.smithi190:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:25:21.964 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:25:21.964 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:21.964 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:25:21.964 DEBUG:teuthology.orchestra.run.smithi190:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:25:21.966 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:25:21.966 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:25:21.966 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:25:21.967 DEBUG:teuthology.orchestra.run.smithi190:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:25:21.968 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:25:21.969 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.128+0000 7f41214d9640 10 monclient: get_monmap_and_config 2024-03-25T18:25:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.128+0000 7f41214d9640 10 monclient: build_initial_monmap 2024-03-25T18:25:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.128+0000 7f41214d9640 10 monclient: monmap: 2024-03-25T18:25:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 0 2024-03-25T18:25:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2024-03-25T18:25:22.129136+0000 2024-03-25T18:25:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:created 2024-03-25T18:25:22.129136+0000 2024-03-25T18:25:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:25:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:25:22.130 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:25:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.129+0000 7f41214d9640 1 Processor -- start 2024-03-25T18:25:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.129+0000 7f41214d9640 1 -- start start 2024-03-25T18:25:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.129+0000 7f41214d9640 10 monclient: init 2024-03-25T18:25:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.130+0000 7f41214d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.130+0000 7f41214d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.130+0000 7f41214d9640 10 monclient(hunting): picked mon.noname-c con 0x7f411c0b4700 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.130+0000 7f41214d9640 10 monclient(hunting): picked mon.noname-b con 0x7f411c142a20 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.130+0000 7f41214d9640 10 monclient(hunting): picked mon.noname-a con 0x7f411c0a96b0 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.130+0000 7f41214d9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c0a9d60 con 0x7f411c0a96b0 2024-03-25T18:25:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.130+0000 7f41214d9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c13ff10 con 0x7f411c0b4700 2024-03-25T18:25:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.130+0000 7f41214d9640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c0aa1f0 con 0x7f411c142a20 2024-03-25T18:25:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.130+0000 7f41214d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.130+0000 7f41214d9640 10 monclient(hunting): authenticate will time out at 7340.298354s 2024-03-25T18:25:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.130+0000 7f411affd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f411c0b4700 0x7f411c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:47032/0 (socket says 172.21.15.99:47032) 2024-03-25T18:25:22.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.130+0000 7f411affd640 1 -- 172.21.15.99:0/2882970994 learned_addr learned my addr 172.21.15.99:0/2882970994 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:25:22.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.134+0000 7f411a7fc640 1 -- 172.21.15.99:0/2882970994 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844719423 0 0) 0x7f411c0a9d60 con 0x7f411c0a96b0 2024-03-25T18:25:22.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.134+0000 7f411a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:22.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.134+0000 7f411a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:22.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.134+0000 7f411a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:22.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.134+0000 7f411a7fc640 10 monclient(hunting): my global_id is 4167 2024-03-25T18:25:22.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.134+0000 7f411a7fc640 1 -- 172.21.15.99:0/2882970994 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4108003520 con 0x7f411c0a96b0 2024-03-25T18:25:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.134+0000 7f411a7fc640 1 -- 172.21.15.99:0/2882970994 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675041507 0 0) 0x7f411c13ff10 con 0x7f411c0b4700 2024-03-25T18:25:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.134+0000 7f411a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.134+0000 7f411a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.134+0000 7f411a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.134+0000 7f411a7fc640 10 monclient(hunting): my global_id is 4177 2024-03-25T18:25:22.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.134+0000 7f411a7fc640 1 -- 172.21.15.99:0/2882970994 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c0a9d60 con 0x7f411c0b4700 2024-03-25T18:25:22.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 1 -- 172.21.15.99:0/2882970994 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1305792831 0 0) 0x7f4108003520 con 0x7f411c0a96b0 2024-03-25T18:25:22.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 1 -- 172.21.15.99:0/2882970994 >> v1:172.21.15.99:6790/0 conn(0x7f411c142a20 legacy=0x7f411c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:25:22.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 1 -- 172.21.15.99:0/2882970994 >> v1:172.21.15.190:6789/0 conn(0x7f411c0b4700 legacy=0x7f411c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:22.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 1 -- 172.21.15.99:0/2882970994 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c0b4ae0 con 0x7f411c0a96b0 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 1 -- 172.21.15.99:0/2882970994 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4114001a10 con 0x7f411c0a96b0 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f411a7fc640 10 monclient: dump: 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:22.142 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:22.142 INFO:teuthology.orchestra.run.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:22.142 INFO:teuthology.orchestra.run.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:22.142 INFO:teuthology.orchestra.run.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:22.142 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:25:22.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f41214d9640 5 monclient: authenticate success, global_id 4167 2024-03-25T18:25:22.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.138+0000 7f41214d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:22.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.142+0000 7f411a7fc640 1 -- 172.21.15.99:0/2882970994 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4114001b70 con 0x7f411c0a96b0 2024-03-25T18:25:22.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.142+0000 7f411a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:22.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.142+0000 7f41214d9640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:22.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.142+0000 7f41214d9640 4 set_mon_vals no callback set 2024-03-25T18:25:22.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.143+0000 7f411a7fc640 1 -- 172.21.15.99:0/2882970994 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4114001f80 con 0x7f411c0a96b0 2024-03-25T18:25:22.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.143+0000 7f411a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.143+0000 7f411a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.143+0000 7f411a7fc640 10 monclient: dump: 2024-03-25T18:25:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2024-03-25T18:25:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:22.144 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:25:22.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.143+0000 7f41214d9640 10 monclient: shutdown 2024-03-25T18:25:22.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.143+0000 7f41214d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:22.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.143+0000 7f41214d9640 1 -- 172.21.15.99:0/2882970994 >> v1:172.21.15.99:6789/0 conn(0x7f411c0a96b0 legacy=0x7f411c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.143+0000 7f41214d9640 1 -- 172.21.15.99:0/2882970994 shutdown_connections 2024-03-25T18:25:22.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.143+0000 7f41214d9640 1 -- 172.21.15.99:0/2882970994 >> 172.21.15.99:0/2882970994 conn(0x7f411c0668e0 msgr2=0x7f411c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:22.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.143+0000 7f41214d9640 1 -- 172.21.15.99:0/2882970994 shutdown_connections 2024-03-25T18:25:22.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.143+0000 7f41214d9640 1 -- 172.21.15.99:0/2882970994 wait complete. 2024-03-25T18:25:22.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.144+0000 7f41214d9640 10 monclient: build_initial_monmap 2024-03-25T18:25:22.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.144+0000 7f41214d9640 10 monclient: monmap: 2024-03-25T18:25:22.146 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 0 2024-03-25T18:25:22.146 INFO:teuthology.orchestra.run.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:22.146 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 0.000000 2024-03-25T18:25:22.146 INFO:teuthology.orchestra.run.smithi099.stderr:created 0.000000 2024-03-25T18:25:22.146 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:22.146 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 1 2024-03-25T18:25:22.146 INFO:teuthology.orchestra.run.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:22.146 INFO:teuthology.orchestra.run.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:22.146 INFO:teuthology.orchestra.run.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:22.146 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.145+0000 7f41214d9640 1 librados: starting msgr at 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.145+0000 7f41214d9640 1 librados: starting objecter 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.145+0000 7f41214d9640 1 Processor -- start 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.145+0000 7f41214d9640 1 -- start start 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.145+0000 7f41214d9640 1 librados: setting wanted keys 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.145+0000 7f41214d9640 1 librados: calling monclient init 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.145+0000 7f41214d9640 10 monclient: init 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.146+0000 7f41214d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.146+0000 7f41214d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.146+0000 7f41214d9640 10 monclient(hunting): picked mon.noname-b con 0x7f411c0a96b0 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.147+0000 7f41214d9640 10 monclient(hunting): picked mon.noname-c con 0x7f411c0b4700 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.147+0000 7f41214d9640 10 monclient(hunting): picked mon.noname-a con 0x7f411c142a20 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.147+0000 7f41214d9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c13c140 con 0x7f411c142a20 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.147+0000 7f41214d9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c1db720 con 0x7f411c0a96b0 2024-03-25T18:25:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.147+0000 7f41214d9640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c1daaf0 con 0x7f411c0b4700 2024-03-25T18:25:22.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.147+0000 7f41214d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:22.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.147+0000 7f411affd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f411c0a96b0 0x7f411c1db010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:47034/0 (socket says 172.21.15.99:47034) 2024-03-25T18:25:22.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.147+0000 7f411affd640 1 -- 172.21.15.99:0/2768309494 learned_addr learned my addr 172.21.15.99:0/2768309494 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:25:22.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.147+0000 7f41214d9640 10 monclient(hunting): authenticate will time out at 7340.315519s 2024-03-25T18:25:22.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.147+0000 7f41137fe640 10 client.?.objecter ms_handle_connect 0x7f411c0a96b0 2024-03-25T18:25:22.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.147+0000 7f41137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:22.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.154+0000 7f41137fe640 10 client.?.objecter ms_handle_connect 0x7f411c0b4700 2024-03-25T18:25:22.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.154+0000 7f41137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:22.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.157+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136913178 0 0) 0x7f411c1daaf0 con 0x7f411c0b4700 2024-03-25T18:25:22.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.157+0000 7f41137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:22.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.157+0000 7f41137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:22.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.157+0000 7f41137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:22.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.157+0000 7f41137fe640 10 monclient(hunting): my global_id is 4175 2024-03-25T18:25:22.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.157+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40fc004eb0 con 0x7f411c0b4700 2024-03-25T18:25:22.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.158+0000 7f41137fe640 10 client.?.objecter ms_handle_connect 0x7f411c142a20 2024-03-25T18:25:22.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.158+0000 7f41137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:22.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.168+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3326249877 0 0) 0x7f40fc004eb0 con 0x7f411c0b4700 2024-03-25T18:25:22.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.168+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f411c1daaf0 con 0x7f411c0b4700 2024-03-25T18:25:22.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.168+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f41040017d0 con 0x7f411c0b4700 2024-03-25T18:25:22.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.168+0000 7f41137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:22.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.168+0000 7f41137fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:22.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.168+0000 7f41137fe640 10 monclient(hunting): dump: 2024-03-25T18:25:22.169 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2024-03-25T18:25:22.169 INFO:teuthology.orchestra.run.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:22.170 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:22.170 INFO:teuthology.orchestra.run.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:22.170 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:22.170 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:22.170 INFO:teuthology.orchestra.run.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:22.170 INFO:teuthology.orchestra.run.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:22.170 INFO:teuthology.orchestra.run.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:22.170 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:25:22.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.168+0000 7f41137fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:22.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1150338217 0 0) 0x7f411c1daaf0 con 0x7f411c0b4700 2024-03-25T18:25:22.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 >> v1:172.21.15.190:6789/0 conn(0x7f411c0a96b0 legacy=0x7f411c1db010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 >> v1:172.21.15.99:6789/0 conn(0x7f411c142a20 legacy=0x7f411c1e2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41137fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:22.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41137fe640 1 monclient: found mon.c 2024-03-25T18:25:22.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:22.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41137fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:22.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c1da7a0 con 0x7f411c0b4700 2024-03-25T18:25:22.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41137fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:22.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41137fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:22.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:22.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41214d9640 5 monclient: authenticate success, global_id 4175 2024-03-25T18:25:22.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41214d9640 10 monclient: _renew_subs 2024-03-25T18:25:22.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41214d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:22.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41214d9640 1 -- 172.21.15.99:0/2768309494 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411c13cb90 con 0x7f411c0b4700 2024-03-25T18:25:22.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41214d9640 10 client.4175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:22.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41214d9640 10 monclient: _renew_subs 2024-03-25T18:25:22.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41214d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:22.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41214d9640 1 -- 172.21.15.99:0/2768309494 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f411c13afb0 con 0x7f411c0b4700 2024-03-25T18:25:22.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.249+0000 7f41214d9640 1 librados: init done 2024-03-25T18:25:22.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.250+0000 7f41214d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:22.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.250+0000 7f41214d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:22.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.250+0000 7f41214d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:22.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.250+0000 7f41214d9640 1 -- 172.21.15.99:0/2768309494 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411c04f730 con 0x7f411c0b4700 2024-03-25T18:25:22.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.360+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4104001d20 con 0x7f411c0b4700 2024-03-25T18:25:22.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.360+0000 7f41137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:22.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.360+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4104003790 con 0x7f411c0b4700 2024-03-25T18:25:22.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.360+0000 7f41137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:22.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.360+0000 7f41137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:22.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.360+0000 7f41137fe640 10 monclient: dump: 2024-03-25T18:25:22.362 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2024-03-25T18:25:22.362 INFO:teuthology.orchestra.run.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:22.362 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:22.362 INFO:teuthology.orchestra.run.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:22.362 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:22.362 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:22.362 INFO:teuthology.orchestra.run.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:22.362 INFO:teuthology.orchestra.run.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:22.362 INFO:teuthology.orchestra.run.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:22.362 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:25:22.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.360+0000 7f4119ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:22.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.360+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1213431838 0 0) 0x7f410400e8f0 con 0x7f411c0b4700 2024-03-25T18:25:22.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.365+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f41040143f0 con 0x7f411c0b4700 2024-03-25T18:25:22.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.365+0000 7f41137fe640 10 client.4175.objecter ms_dispatch 0x7f411c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:22.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.365+0000 7f41137fe640 3 client.4175.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:22.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.365+0000 7f41137fe640 3 client.4175.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:22.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.365+0000 7f41137fe640 20 client.4175.objecter dump_active .. 0 homeless 2024-03-25T18:25:22.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.365+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f41040136b0 con 0x7f411c0b4700 2024-03-25T18:25:22.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.365+0000 7f41137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:22.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.365+0000 7f41137fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:22.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.470+0000 7f41214d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:22.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.470+0000 7f41214d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:22.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.470+0000 7f41214d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:22.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.470+0000 7f41214d9640 1 -- 172.21.15.99:0/2768309494 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f411c069f30 con 0x7f411c0b4700 2024-03-25T18:25:22.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.593+0000 7f41137fe640 1 -- 172.21.15.99:0/2768309494 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f410401c3d0 con 0x7f411c0b4700 2024-03-25T18:25:22.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.593+0000 7f41137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:22.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.593+0000 7f41137fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:22.595 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-25T18:25:22.595 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:22.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.595+0000 7f41214d9640 10 librados: watch_flush enter 2024-03-25T18:25:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.595+0000 7f41214d9640 10 librados: watch_flush exit 2024-03-25T18:25:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.595+0000 7f41214d9640 20 client.4175.objecter shutdown clearing up homeless session... 2024-03-25T18:25:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.595+0000 7f41214d9640 10 client.4175.objecter successfully canceled tick 2024-03-25T18:25:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.595+0000 7f41214d9640 10 monclient: shutdown 2024-03-25T18:25:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.595+0000 7f41214d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.595+0000 7f41214d9640 1 -- 172.21.15.99:0/2768309494 >> v1:172.21.15.99:6790/0 conn(0x7f411c0b4700 legacy=0x7f411c133f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.595+0000 7f41214d9640 1 -- 172.21.15.99:0/2768309494 shutdown_connections 2024-03-25T18:25:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.595+0000 7f41214d9640 1 -- 172.21.15.99:0/2768309494 >> 172.21.15.99:0/2768309494 conn(0x7f411c0668e0 msgr2=0x7f411c066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.595+0000 7f41214d9640 1 -- 172.21.15.99:0/2768309494 shutdown_connections 2024-03-25T18:25:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.595+0000 7f41214d9640 1 -- 172.21.15.99:0/2768309494 wait complete. 2024-03-25T18:25:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-25T18:25:22.595+0000 7f41214d9640 1 librados: shutdown 2024-03-25T18:25:22.611 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:25:22.611 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:25:22.919 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:22.918+0000 7f75bcce9640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:25:23.176 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:25:23.175+0000 7f75bcce9640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:25:23.418 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:23.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.560+0000 7f0381663640 10 monclient: get_monmap_and_config 2024-03-25T18:25:23.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.560+0000 7f0381663640 10 monclient: build_initial_monmap 2024-03-25T18:25:23.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.560+0000 7f0381663640 10 monclient: monmap: 2024-03-25T18:25:23.561 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:25:23.562 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:23.562 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:25:23.561619+0000 2024-03-25T18:25:23.562 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:25:23.561619+0000 2024-03-25T18:25:23.562 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:23.562 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:23.562 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:23.562 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:25:23.562 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:25:23.562 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:23.564 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.561+0000 7f0381663640 1 Processor -- start 2024-03-25T18:25:23.564 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.561+0000 7f0381663640 1 -- start start 2024-03-25T18:25:23.564 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.561+0000 7f0381663640 10 monclient: init 2024-03-25T18:25:23.564 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.562+0000 7f0381663640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:23.564 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.562+0000 7f0381663640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:23.564 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.562+0000 7f0381663640 10 monclient(hunting): picked mon.noname-b con 0x7f037c0a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:23.564 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.562+0000 7f0381663640 10 monclient(hunting): picked mon.noname-a con 0x7f037c13f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:23.564 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.562+0000 7f0381663640 10 monclient(hunting): picked mon.noname-c con 0x7f037c1430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:23.564 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.563+0000 7f0381663640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037c142520 con 0x7f037c13f710 2024-03-25T18:25:23.564 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.563+0000 7f0381663640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037c0b28a0 con 0x7f037c1430c0 2024-03-25T18:25:23.564 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.563+0000 7f0381663640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037c0b2a30 con 0x7f037c0a98f0 2024-03-25T18:25:23.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.563+0000 7f0381663640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:23.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.563+0000 7f0381663640 10 monclient(hunting): authenticate will time out at 7341.731028s 2024-03-25T18:25:23.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.564+0000 7f037affd640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f037c0a98f0 0x7f037c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:54740/0 (socket says 172.21.15.99:54740) 2024-03-25T18:25:23.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.564+0000 7f037affd640 1 -- 172.21.15.99:0/2405210942 learned_addr learned my addr 172.21.15.99:0/2405210942 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:25:23.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.576+0000 7f037a7fc640 1 -- 172.21.15.99:0/2405210942 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018493441 0 0) 0x7f037c0b28a0 con 0x7f037c1430c0 2024-03-25T18:25:23.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.576+0000 7f037a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:23.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.576+0000 7f037a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:23.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.576+0000 7f037a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:23.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.576+0000 7f037a7fc640 10 monclient(hunting): my global_id is 4183 2024-03-25T18:25:23.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.576+0000 7f037a7fc640 1 -- 172.21.15.99:0/2405210942 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0368003520 con 0x7f037c1430c0 2024-03-25T18:25:23.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.576+0000 7f037a7fc640 1 -- 172.21.15.99:0/2405210942 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683547384 0 0) 0x7f037c0b2a30 con 0x7f037c0a98f0 2024-03-25T18:25:23.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.576+0000 7f037a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:23.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.576+0000 7f037a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:23.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.576+0000 7f037a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:23.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.576+0000 7f037a7fc640 10 monclient(hunting): my global_id is 4178 2024-03-25T18:25:23.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.576+0000 7f037a7fc640 1 -- 172.21.15.99:0/2405210942 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f037c0b28a0 con 0x7f037c0a98f0 2024-03-25T18:25:23.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.578+0000 7f037a7fc640 1 -- 172.21.15.99:0/2405210942 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253486444 0 0) 0x7f037c142520 con 0x7f037c13f710 2024-03-25T18:25:23.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.578+0000 7f037a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:23.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.578+0000 7f037a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:23.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.578+0000 7f037a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:23.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.578+0000 7f037a7fc640 10 monclient(hunting): my global_id is 4170 2024-03-25T18:25:23.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.578+0000 7f037a7fc640 1 -- 172.21.15.99:0/2405210942 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f037c0b2a30 con 0x7f037c13f710 2024-03-25T18:25:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f037a7fc640 1 -- 172.21.15.99:0/2405210942 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2744209223 0 0) 0x7f0368003520 con 0x7f037c1430c0 2024-03-25T18:25:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f037a7fc640 1 -- 172.21.15.99:0/2405210942 >> v1:172.21.15.99:6790/0 conn(0x7f037c0a98f0 legacy=0x7f037c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f037a7fc640 1 -- 172.21.15.99:0/2405210942 >> v1:172.21.15.99:6789/0 conn(0x7f037c13f710 legacy=0x7f037c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f037a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f037a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:25:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f037a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f037a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:25:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f037a7fc640 1 -- 172.21.15.99:0/2405210942 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c0b44c0 con 0x7f037c1430c0 2024-03-25T18:25:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f037a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:23.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f037a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:23.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f037a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:23.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f037a7fc640 1 -- 172.21.15.99:0/2405210942 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0374001a10 con 0x7f037c1430c0 2024-03-25T18:25:23.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f0381663640 5 monclient: authenticate success, global_id 4183 2024-03-25T18:25:23.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f0381663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:23.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.582+0000 7f037a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:23.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.583+0000 7f037a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:23.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.583+0000 7f037a7fc640 10 monclient: dump: 2024-03-25T18:25:23.585 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:23.585 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:23.585 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:23.585 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:23.585 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:23.585 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:23.586 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:23.586 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:23.586 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:23.586 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:23.586 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.583+0000 7f0381663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:23.587 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.584+0000 7f037a7fc640 1 -- 172.21.15.99:0/2405210942 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0374001c30 con 0x7f037c1430c0 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.584+0000 7f037a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.584+0000 7f0381663640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.584+0000 7f0381663640 4 set_mon_vals no callback set 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.585+0000 7f037a7fc640 1 -- 172.21.15.99:0/2405210942 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0374001fc0 con 0x7f037c1430c0 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.585+0000 7f037a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.585+0000 7f037a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.585+0000 7f037a7fc640 10 monclient: dump: 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:23.588 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:23.589 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:23.589 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:23.589 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.585+0000 7f0381663640 10 monclient: shutdown 2024-03-25T18:25:23.589 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.585+0000 7f0381663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:23.589 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.585+0000 7f0381663640 1 -- 172.21.15.99:0/2405210942 >> v1:172.21.15.190:6789/0 conn(0x7f037c1430c0 legacy=0x7f037c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:23.589 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.585+0000 7f0381663640 1 -- 172.21.15.99:0/2405210942 shutdown_connections 2024-03-25T18:25:23.589 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.586+0000 7f0381663640 1 -- 172.21.15.99:0/2405210942 >> 172.21.15.99:0/2405210942 conn(0x7f037c0b7ea0 msgr2=0x7f037c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:23.589 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.586+0000 7f0381663640 1 -- 172.21.15.99:0/2405210942 shutdown_connections 2024-03-25T18:25:23.589 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.586+0000 7f0381663640 1 -- 172.21.15.99:0/2405210942 wait complete. 2024-03-25T18:25:23.589 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.587+0000 7f0381663640 10 monclient: build_initial_monmap 2024-03-25T18:25:23.589 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.587+0000 7f0381663640 10 monclient: monmap: 2024-03-25T18:25:23.589 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:25:23.589 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:23.590 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:25:23.590 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:25:23.590 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:23.590 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:25:23.590 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:23.590 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:23.590 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:23.590 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:23.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.587+0000 7f0381663640 1 librados: starting msgr at 2024-03-25T18:25:23.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.587+0000 7f0381663640 1 librados: starting objecter 2024-03-25T18:25:23.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.587+0000 7f0381663640 1 Processor -- start 2024-03-25T18:25:23.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.587+0000 7f0381663640 1 -- start start 2024-03-25T18:25:23.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.587+0000 7f0381663640 1 librados: setting wanted keys 2024-03-25T18:25:23.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.588+0000 7f0381663640 1 librados: calling monclient init 2024-03-25T18:25:23.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.588+0000 7f0381663640 10 monclient: init 2024-03-25T18:25:23.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.588+0000 7f0381663640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:23.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.589+0000 7f0381663640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:23.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.589+0000 7f0381663640 10 monclient(hunting): picked mon.noname-c con 0x7f037c0a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:23.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.589+0000 7f0381663640 10 monclient(hunting): picked mon.noname-a con 0x7f037c13f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:23.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.589+0000 7f0381663640 10 monclient(hunting): picked mon.noname-b con 0x7f037c1430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:23.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.589+0000 7f0381663640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037c0abdc0 con 0x7f037c13f710 2024-03-25T18:25:23.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.589+0000 7f0381663640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037c1d41e0 con 0x7f037c1430c0 2024-03-25T18:25:23.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.589+0000 7f0381663640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037c1d4430 con 0x7f037c0a98f0 2024-03-25T18:25:23.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.589+0000 7f0381663640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:23.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.589+0000 7f0381663640 10 monclient(hunting): authenticate will time out at 7341.757352s 2024-03-25T18:25:23.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.589+0000 7f0373fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f037c13f710 0x7f037c1d4950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:48592/0 (socket says 172.21.15.99:48592) 2024-03-25T18:25:23.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.589+0000 7f0373fff640 1 -- 172.21.15.99:0/3146645164 learned_addr learned my addr 172.21.15.99:0/3146645164 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:25:23.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.590+0000 7f03737fe640 10 client.?.objecter ms_handle_connect 0x7f037c13f710 2024-03-25T18:25:23.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.590+0000 7f03737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:23.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.590+0000 7f03737fe640 10 client.?.objecter ms_handle_connect 0x7f037c1430c0 2024-03-25T18:25:23.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.590+0000 7f03737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:23.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.591+0000 7f03737fe640 10 client.?.objecter ms_handle_connect 0x7f037c0a98f0 2024-03-25T18:25:23.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.591+0000 7f03737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.592+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135897466 0 0) 0x7f037c0abdc0 con 0x7f037c13f710 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.592+0000 7f03737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.592+0000 7f03737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.592+0000 7f03737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.592+0000 7f03737fe640 10 monclient(hunting): my global_id is 4173 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.592+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f035c004eb0 con 0x7f037c13f710 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.593+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162750858 0 0) 0x7f037c1d41e0 con 0x7f037c1430c0 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.593+0000 7f03737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.593+0000 7f03737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.593+0000 7f03737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.593+0000 7f03737fe640 10 monclient(hunting): my global_id is 4186 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.593+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f037c0abdc0 con 0x7f037c1430c0 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.595+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953628581 0 0) 0x7f037c1d4430 con 0x7f037c0a98f0 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.595+0000 7f03737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:23.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.595+0000 7f03737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:23.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.595+0000 7f03737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:23.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.595+0000 7f03737fe640 10 monclient(hunting): my global_id is 4181 2024-03-25T18:25:23.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.595+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f037c1d41e0 con 0x7f037c0a98f0 2024-03-25T18:25:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.600+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1693636058 0 0) 0x7f035c004eb0 con 0x7f037c13f710 2024-03-25T18:25:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.601+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f037c1d4430 con 0x7f037c13f710 2024-03-25T18:25:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.601+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4143009658 0 0) 0x7f037c0abdc0 con 0x7f037c1430c0 2024-03-25T18:25:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.601+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f035c004eb0 con 0x7f037c1430c0 2024-03-25T18:25:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.601+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0364001a10 con 0x7f037c13f710 2024-03-25T18:25:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.601+0000 7f03737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.601+0000 7f03737fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.601+0000 7f03737fe640 10 monclient(hunting): dump: 2024-03-25T18:25:23.602 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:23.602 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:23.602 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:23.602 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:23.602 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.601+0000 7f03737fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.601+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0374001da0 con 0x7f037c1430c0 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.601+0000 7f03737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.601+0000 7f03737fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.601+0000 7f03737fe640 10 monclient(hunting): dump: 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:23.603 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:23.604 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:23.604 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:23.604 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:23.604 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:23.604 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:23.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.602+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2050300646 0 0) 0x7f037c1d41e0 con 0x7f037c0a98f0 2024-03-25T18:25:23.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.602+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f037c0abdc0 con 0x7f037c0a98f0 2024-03-25T18:25:23.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.602+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f036c001930 con 0x7f037c0a98f0 2024-03-25T18:25:23.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.602+0000 7f03737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:23.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.602+0000 7f03737fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:23.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.602+0000 7f03737fe640 10 monclient(hunting): dump: 2024-03-25T18:25:23.605 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:23.605 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:23.605 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:23.605 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:23.605 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:23.605 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:23.605 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:23.605 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:23.605 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:23.605 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:23.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 24113457 0 0) 0x7f037c1d4430 con 0x7f037c13f710 2024-03-25T18:25:23.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 >> v1:172.21.15.99:6790/0 conn(0x7f037c0a98f0 legacy=0x7f037c138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:23.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 >> v1:172.21.15.190:6789/0 conn(0x7f037c1430c0 legacy=0x7f037c134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:23.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f03737fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:23.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f03737fe640 1 monclient: found mon.a 2024-03-25T18:25:23.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f03737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:23.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f03737fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:23.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c1391a0 con 0x7f037c13f710 2024-03-25T18:25:23.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f03737fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:23.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f03737fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:23.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f03737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:23.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f0381663640 5 monclient: authenticate success, global_id 4173 2024-03-25T18:25:23.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f0381663640 10 monclient: _renew_subs 2024-03-25T18:25:23.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f0381663640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:23.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f0381663640 1 -- 172.21.15.99:0/3146645164 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037c1d5060 con 0x7f037c13f710 2024-03-25T18:25:23.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f0381663640 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:23.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f0381663640 10 monclient: _renew_subs 2024-03-25T18:25:23.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f0381663640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:23.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f0381663640 1 -- 172.21.15.99:0/3146645164 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f037c138520 con 0x7f037c13f710 2024-03-25T18:25:23.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.606+0000 7f0381663640 1 librados: init done 2024-03-25T18:25:23.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.607+0000 7f0381663640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:23.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.607+0000 7f0381663640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:23.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.607+0000 7f0381663640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:23.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.607+0000 7f0381663640 1 -- 172.21.15.99:0/3146645164 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f037c14f320 con 0x7f037c13f710 2024-03-25T18:25:23.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.828+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0364001c70 con 0x7f037c13f710 2024-03-25T18:25:23.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.828+0000 7f03737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:23.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.828+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0364003a60 con 0x7f037c13f710 2024-03-25T18:25:23.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.828+0000 7f03737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:23.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.828+0000 7f03737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:23.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.828+0000 7f03737fe640 10 monclient: dump: 2024-03-25T18:25:23.828 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:23.828 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:23.828 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:23.828 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:23.828 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:23.828 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:23.828 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:23.829 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:23.829 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:23.829 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:23.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:23.828+0000 7f0379ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:24.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.049+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 3543237385 0 0) 0x7f0364003bc0 con 0x7f037c13f710 2024-03-25T18:25:24.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.050+0000 7f037affd640 10 monclient: get_auth_request con 0x7f035c06dff0 auth_method 0 2024-03-25T18:25:24.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.050+0000 7f03737fe640 10 client.4173.objecter ms_handle_connect 0x7f035c06dff0 2024-03-25T18:25:24.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.059+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f0364054830 con 0x7f037c13f710 2024-03-25T18:25:24.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.059+0000 7f03737fe640 10 client.4173.objecter ms_dispatch 0x7f037c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:24.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.059+0000 7f03737fe640 3 client.4173.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:24.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.059+0000 7f03737fe640 3 client.4173.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:24.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.059+0000 7f03737fe640 20 client.4173.objecter dump_active .. 0 homeless 2024-03-25T18:25:24.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.059+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0364050340 con 0x7f037c13f710 2024-03-25T18:25:24.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.059+0000 7f03737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:24.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.059+0000 7f03737fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:24.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.412+0000 7f0381663640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:24.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.412+0000 7f0381663640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:24.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.412+0000 7f0381663640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:24.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.412+0000 7f0381663640 1 -- 172.21.15.99:0/3146645164 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f037c1d45c0 con 0x7f037c13f710 2024-03-25T18:25:24.618 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.617+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f036404fdf0 con 0x7f037c13f710 2024-03-25T18:25:24.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.694+0000 7f03737fe640 1 -- 172.21.15.99:0/3146645164 <== mon.0 v1:172.21.15.99:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f03640038c0 con 0x7f037c13f710 2024-03-25T18:25:24.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.694+0000 7f03737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:24.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.694+0000 7f03737fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:24.696 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:25:24.696 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:24.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.697+0000 7f0381663640 10 librados: watch_flush enter 2024-03-25T18:25:24.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.697+0000 7f0381663640 10 librados: watch_flush exit 2024-03-25T18:25:24.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.697+0000 7f0381663640 20 client.4173.objecter shutdown clearing up homeless session... 2024-03-25T18:25:24.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.697+0000 7f0381663640 10 client.4173.objecter successfully canceled tick 2024-03-25T18:25:24.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.697+0000 7f0381663640 1 -- 172.21.15.99:0/3146645164 >> v1:172.21.15.190:6800/4220409699 conn(0x7f035c06dff0 legacy=0x7f035c0704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.697+0000 7f0381663640 10 monclient: shutdown 2024-03-25T18:25:24.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.697+0000 7f0381663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:24.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.697+0000 7f0381663640 1 -- 172.21.15.99:0/3146645164 >> v1:172.21.15.99:6789/0 conn(0x7f037c13f710 legacy=0x7f037c1d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.697+0000 7f0381663640 1 -- 172.21.15.99:0/3146645164 shutdown_connections 2024-03-25T18:25:24.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.697+0000 7f0381663640 1 -- 172.21.15.99:0/3146645164 >> 172.21.15.99:0/3146645164 conn(0x7f037c0b7ea0 msgr2=0x7f037c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:24.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.697+0000 7f0381663640 1 -- 172.21.15.99:0/3146645164 shutdown_connections 2024-03-25T18:25:24.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.697+0000 7f0381663640 1 -- 172.21.15.99:0/3146645164 wait complete. 2024-03-25T18:25:24.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:24.697+0000 7f0381663640 1 librados: shutdown 2024-03-25T18:25:24.713 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:28.763 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:06.730 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:28.824 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:06.791 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:28.836 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:06.804 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:28.896 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:06.864 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:28.905 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:06.873 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:28.965 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:06.924 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:28.966 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:06.933 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:29.023 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:06.982 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:29.111 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:07.102 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:29.118 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:07.122 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:29.179 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:07.183 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:29.193 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:07.184 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:29.319 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:07.299 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:29.371 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-03-25T18:25:29.365+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:29.378 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:07.358 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:29.447 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-03-25T18:25:29.440+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:29.514 INFO:tasks.ceph.osd.6.smithi190.stderr:2024-03-25T18:25:29.508+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:29.567 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:07.557 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:29.577 INFO:tasks.ceph.osd.7.smithi190.stderr:2024-03-25T18:25:29.570+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:29.626 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:07.616 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:29.745 INFO:tasks.ceph.osd.2.smithi099.stderr:2024-03-25T18:25:29.738+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:29.747 INFO:tasks.ceph.osd.0.smithi099.stderr:2024-03-25T18:25:29.734+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:29.832 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:07.799 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:29.910 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:07.878 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:29.939 INFO:tasks.ceph.osd.3.smithi099.stderr:2024-03-25T18:25:29.933+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:29.978 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:07.946 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.044 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:08.003 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.196 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:08.164 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.197 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:08.188 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.203 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:08.207 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.224 INFO:tasks.ceph.osd.1.smithi099.stderr:2024-03-25T18:25:30.216+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:25:30.276 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:08.244 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.347 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:08.314 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.421 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:08.380 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.426 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:08.406 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.513 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:08.481 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.565 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:08.556 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.583 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:08.588 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.636 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:08.604 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.679 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:08.670 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.705 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:08.673 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.775 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:08.734 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.796 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:08.763 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.804 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:08.784 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.878 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:08.868 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.928 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:08.896 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.941 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:08.946 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:30.994 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:08.961 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.045 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:09.035 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.064 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:09.031 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.073 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:09.032 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.143 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:09.134 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.158 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:09.138 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.219 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:09.187 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.234 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:09.238 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.285 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:09.252 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.339 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:09.307 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.359 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:09.350 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.363 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:09.322 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.407 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:09.398 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.454 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:09.435 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.484 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:09.452 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.518 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:31.522 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:09.527 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.573 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:09.540 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.608 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.607+0000 7f85199a8640 10 monclient: get_monmap_and_config 2024-03-25T18:25:31.608 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.607+0000 7f85199a8640 10 monclient: build_initial_monmap 2024-03-25T18:25:31.608 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.607+0000 7f85199a8640 10 monclient: monmap: 2024-03-25T18:25:31.608 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:25:31.608 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:31.608 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:25:31.608219+0000 2024-03-25T18:25:31.608 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:25:31.608219+0000 2024-03-25T18:25:31.608 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:31.608 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:31.608 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:31.608 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:25:31.609 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:25:31.609 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:31.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.607+0000 7f85199a8640 1 Processor -- start 2024-03-25T18:25:31.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.607+0000 7f85199a8640 1 -- start start 2024-03-25T18:25:31.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.607+0000 7f85199a8640 10 monclient: init 2024-03-25T18:25:31.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.608+0000 7f85199a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:31.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.608+0000 7f85199a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:31.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.608+0000 7f85199a8640 10 monclient(hunting): picked mon.noname-a con 0x7f85140a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:31.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.608+0000 7f85199a8640 10 monclient(hunting): picked mon.noname-b con 0x7f851413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:31.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.608+0000 7f85199a8640 10 monclient(hunting): picked mon.noname-c con 0x7f85141430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:31.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.608+0000 7f85199a8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8514142520 con 0x7f85140a98f0 2024-03-25T18:25:31.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.608+0000 7f85199a8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85140b28a0 con 0x7f85141430c0 2024-03-25T18:25:31.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.608+0000 7f85199a8640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85140b2a30 con 0x7f851413f710 2024-03-25T18:25:31.611 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.608+0000 7f85199a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:31.611 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.608+0000 7f85199a8640 10 monclient(hunting): authenticate will time out at 7349.776304s 2024-03-25T18:25:31.611 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.608+0000 7f8512ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f85140a98f0 0x7f85140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:35246/0 (socket says 172.21.15.99:35246) 2024-03-25T18:25:31.611 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.608+0000 7f8512ffd640 1 -- 172.21.15.99:0/3389706170 learned_addr learned my addr 172.21.15.99:0/3389706170 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:25:31.612 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 1 -- 172.21.15.99:0/3389706170 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014134980 0 0) 0x7f85140b2a30 con 0x7f851413f710 2024-03-25T18:25:31.612 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.612 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.612 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.612 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:25:31.612 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 1 -- 172.21.15.99:0/3389706170 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84fc003520 con 0x7f851413f710 2024-03-25T18:25:31.612 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 1 -- 172.21.15.99:0/3389706170 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191833805 0 0) 0x7f8514142520 con 0x7f85140a98f0 2024-03-25T18:25:31.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 10 monclient(hunting): my global_id is 4200 2024-03-25T18:25:31.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 1 -- 172.21.15.99:0/3389706170 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85140b2a30 con 0x7f85140a98f0 2024-03-25T18:25:31.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 1 -- 172.21.15.99:0/3389706170 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979980244 0 0) 0x7f85140b28a0 con 0x7f85141430c0 2024-03-25T18:25:31.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:25:31.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.611+0000 7f8511ffb640 1 -- 172.21.15.99:0/3389706170 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8514142520 con 0x7f85141430c0 2024-03-25T18:25:31.614 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:09.580 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 1 -- 172.21.15.99:0/3389706170 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1573377480 0 0) 0x7f84fc003520 con 0x7f851413f710 2024-03-25T18:25:31.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 1 -- 172.21.15.99:0/3389706170 >> v1:172.21.15.190:6789/0 conn(0x7f85141430c0 legacy=0x7f851414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 1 -- 172.21.15.99:0/3389706170 >> v1:172.21.15.99:6789/0 conn(0x7f85140a98f0 legacy=0x7f85140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:31.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 1 monclient: found mon.noname-b 2024-03-25T18:25:31.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:31.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 1 -- 172.21.15.99:0/3389706170 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85140b44c0 con 0x7f851413f710 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 1 -- 172.21.15.99:0/3389706170 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8508001690 con 0x7f851413f710 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f85199a8640 5 monclient: authenticate success, global_id 4208 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f85199a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f8511ffb640 10 monclient: dump: 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.616 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:31.617 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:31.617 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:31.617 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:31.617 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:31.617 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.614+0000 7f85199a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:31.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.615+0000 7f8511ffb640 1 -- 172.21.15.99:0/3389706170 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85080017f0 con 0x7f851413f710 2024-03-25T18:25:31.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.615+0000 7f8511ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:31.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.615+0000 7f8511ffb640 1 -- 172.21.15.99:0/3389706170 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8508001c00 con 0x7f851413f710 2024-03-25T18:25:31.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.615+0000 7f8511ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.616+0000 7f8511ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:31.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.616+0000 7f8511ffb640 10 monclient: dump: 2024-03-25T18:25:31.620 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:31.620 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:31.620 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:31.620 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:31.620 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.620 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:31.620 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:31.620 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:31.620 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.616+0000 7f85199a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.616+0000 7f85199a8640 4 set_mon_vals no callback set 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.616+0000 7f85199a8640 10 monclient: shutdown 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.616+0000 7f85199a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.616+0000 7f85199a8640 1 -- 172.21.15.99:0/3389706170 >> v1:172.21.15.99:6790/0 conn(0x7f851413f710 legacy=0x7f85141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.616+0000 7f85199a8640 1 -- 172.21.15.99:0/3389706170 shutdown_connections 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.616+0000 7f85199a8640 1 -- 172.21.15.99:0/3389706170 >> 172.21.15.99:0/3389706170 conn(0x7f85140b7ea0 msgr2=0x7f85140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.616+0000 7f85199a8640 1 -- 172.21.15.99:0/3389706170 shutdown_connections 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.616+0000 7f85199a8640 1 -- 172.21.15.99:0/3389706170 wait complete. 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.617+0000 7f85199a8640 10 monclient: build_initial_monmap 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.617+0000 7f85199a8640 10 monclient: monmap: 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:31.621 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.617+0000 7f85199a8640 1 librados: starting msgr at 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.617+0000 7f85199a8640 1 librados: starting objecter 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.617+0000 7f85199a8640 1 Processor -- start 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.617+0000 7f85199a8640 1 -- start start 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.617+0000 7f85199a8640 1 librados: setting wanted keys 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.617+0000 7f85199a8640 1 librados: calling monclient init 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.617+0000 7f85199a8640 10 monclient: init 2024-03-25T18:25:31.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.618+0000 7f85199a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:31.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.618+0000 7f85199a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:31.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.618+0000 7f85199a8640 10 monclient(hunting): picked mon.noname-c con 0x7f85140a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:31.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.618+0000 7f85199a8640 10 monclient(hunting): picked mon.noname-a con 0x7f851413f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:31.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.618+0000 7f85199a8640 10 monclient(hunting): picked mon.noname-b con 0x7f85141430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:31.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.618+0000 7f85199a8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8514138730 con 0x7f851413f710 2024-03-25T18:25:31.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.618+0000 7f85199a8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8514138a00 con 0x7f85141430c0 2024-03-25T18:25:31.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.618+0000 7f85199a8640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85140b1790 con 0x7f85140a98f0 2024-03-25T18:25:31.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.618+0000 7f85199a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:31.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.618+0000 7f85199a8640 10 monclient(hunting): authenticate will time out at 7349.786335s 2024-03-25T18:25:31.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.618+0000 7f85127fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f851413f710 0x7f8514134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:35248/0 (socket says 172.21.15.99:35248) 2024-03-25T18:25:31.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.618+0000 7f85127fc640 1 -- 172.21.15.99:0/2996302852 learned_addr learned my addr 172.21.15.99:0/2996302852 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:25:31.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.619+0000 7f84fbfff640 10 client.?.objecter ms_handle_connect 0x7f851413f710 2024-03-25T18:25:31.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.619+0000 7f84fbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:31.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.619+0000 7f84fbfff640 10 client.?.objecter ms_handle_connect 0x7f85140a98f0 2024-03-25T18:25:31.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.619+0000 7f84fbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:31.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.620+0000 7f84fbfff640 10 client.?.objecter ms_handle_connect 0x7f85141430c0 2024-03-25T18:25:31.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.620+0000 7f84fbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:31.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.620+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241325517 0 0) 0x7f8514138730 con 0x7f851413f710 2024-03-25T18:25:31.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.620+0000 7f84fbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.620+0000 7f84fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.620+0000 7f84fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.620+0000 7f84fbfff640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:25:31.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.621+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8500004eb0 con 0x7f851413f710 2024-03-25T18:25:31.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.621+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186263085 0 0) 0x7f85140b1790 con 0x7f85140a98f0 2024-03-25T18:25:31.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.621+0000 7f84fbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.621+0000 7f84fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.621+0000 7f84fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.621+0000 7f84fbfff640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:25:31.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.621+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8514138730 con 0x7f85140a98f0 2024-03-25T18:25:31.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.622+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399678932 0 0) 0x7f8514138a00 con 0x7f85141430c0 2024-03-25T18:25:31.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.622+0000 7f84fbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.622+0000 7f84fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.622+0000 7f84fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.622+0000 7f84fbfff640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:25:31.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.622+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85140b1790 con 0x7f85141430c0 2024-03-25T18:25:31.627 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.625+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2704414122 0 0) 0x7f8500004eb0 con 0x7f851413f710 2024-03-25T18:25:31.627 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.625+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8514138a00 con 0x7f851413f710 2024-03-25T18:25:31.627 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.625+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8508001bc0 con 0x7f851413f710 2024-03-25T18:25:31.627 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.625+0000 7f84fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.627 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.625+0000 7f84fbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:31.627 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.625+0000 7f84fbfff640 10 monclient(hunting): dump: 2024-03-25T18:25:31.627 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:31.628 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:31.628 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:31.628 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:31.628 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.628 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:31.628 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:31.628 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:31.628 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:31.628 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:31.628 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.625+0000 7f84fbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:31.628 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.625+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3564577622 0 0) 0x7f8514138730 con 0x7f85140a98f0 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.625+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8500004eb0 con 0x7f85140a98f0 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.625+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8504001c70 con 0x7f85140a98f0 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.625+0000 7f84fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.625+0000 7f84fbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.625+0000 7f84fbfff640 10 monclient(hunting): dump: 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:31.629 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:31.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.627+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 542476084 0 0) 0x7f85140b1790 con 0x7f85141430c0 2024-03-25T18:25:31.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.627+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8514138730 con 0x7f85141430c0 2024-03-25T18:25:31.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.627+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f850c001850 con 0x7f85141430c0 2024-03-25T18:25:31.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.627+0000 7f84fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.627+0000 7f84fbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:31.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.627+0000 7f84fbfff640 10 monclient(hunting): dump: 2024-03-25T18:25:31.632 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:31.632 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:31.632 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:31.632 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:31.632 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.632 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:31.632 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:31.632 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:31.632 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2941482798 0 0) 0x7f8514138a00 con 0x7f851413f710 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 >> v1:172.21.15.99:6790/0 conn(0x7f85140a98f0 legacy=0x7f85141379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 >> v1:172.21.15.190:6789/0 conn(0x7f85141430c0 legacy=0x7f85141da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f84fbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f84fbfff640 1 monclient: found mon.a 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f84fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f84fbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85140b12e0 con 0x7f851413f710 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f84fbfff640 10 monclient: _finish_auth 0 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f84fbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f84fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f85199a8640 5 monclient: authenticate success, global_id 4203 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f85199a8640 10 monclient: _renew_subs 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f85199a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:31.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.628+0000 7f85199a8640 1 -- 172.21.15.99:0/2996302852 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85140b1530 con 0x7f851413f710 2024-03-25T18:25:31.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.629+0000 7f85199a8640 10 client.4203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:31.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.629+0000 7f85199a8640 10 monclient: _renew_subs 2024-03-25T18:25:31.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.629+0000 7f85199a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:31.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.629+0000 7f85199a8640 1 -- 172.21.15.99:0/2996302852 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8514134900 con 0x7f851413f710 2024-03-25T18:25:31.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.629+0000 7f85199a8640 1 librados: init done 2024-03-25T18:25:31.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.629+0000 7f85199a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:31.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.629+0000 7f85199a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:31.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.629+0000 7f85199a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:31.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.629+0000 7f85199a8640 1 -- 172.21.15.99:0/2996302852 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85140627b0 con 0x7f851413f710 2024-03-25T18:25:31.634 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:09.621 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.644+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85080011f0 con 0x7f851413f710 2024-03-25T18:25:31.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.644+0000 7f84fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:31.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.644+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8508003b00 con 0x7f851413f710 2024-03-25T18:25:31.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.644+0000 7f84fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.644+0000 7f84fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:31.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.644+0000 7f84fbfff640 10 monclient: dump: 2024-03-25T18:25:31.645 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:31.645 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:31.645 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:31.646 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:31.646 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.646 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:31.646 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:31.646 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:31.646 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:31.646 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:31.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.644+0000 7f85117fa640 4 set_mon_vals no callback set 2024-03-25T18:25:31.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.645+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f8508003d20 con 0x7f851413f710 2024-03-25T18:25:31.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.645+0000 7f8512ffd640 10 monclient: get_auth_request con 0x7f850006e0a0 auth_method 0 2024-03-25T18:25:31.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.645+0000 7f84fbfff640 10 client.4203.objecter ms_handle_connect 0x7f850006e0a0 2024-03-25T18:25:31.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.646+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f85080548c0 con 0x7f851413f710 2024-03-25T18:25:31.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.646+0000 7f84fbfff640 10 client.4203.objecter ms_dispatch 0x7f8514000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:31.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.646+0000 7f84fbfff640 3 client.4203.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:31.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.646+0000 7f84fbfff640 3 client.4203.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:31.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.646+0000 7f84fbfff640 20 client.4203.objecter dump_active .. 0 homeless 2024-03-25T18:25:31.651 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:09.610 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.674 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:09.665 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.748 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:09.728 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.771 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:09.739 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.818 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:09.822 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.861 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:09.828 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.878 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.877+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85080503d0 con 0x7f851413f710 2024-03-25T18:25:31.878 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.877+0000 7f84fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:31.878 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:31.877+0000 7f84fbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:31.886 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:09.853 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.901 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:09.891 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.939 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:09.898 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:31.941 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:09.932 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.037 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:10.017 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.059 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:10.027 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.070+0000 7f85199a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:32.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.070+0000 7f85199a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:32.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.070+0000 7f85199a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:32.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.070+0000 7f85199a8640 1 -- 172.21.15.99:0/2996302852 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8514069f40 con 0x7f851413f710 2024-03-25T18:25:32.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.084+0000 7f84fbfff640 1 -- 172.21.15.99:0/2996302852 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f8508052300 con 0x7f851413f710 2024-03-25T18:25:32.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.084+0000 7f84fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:32.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.084+0000 7f84fbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:32.085 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:25:32.085 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:32.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.086+0000 7f85199a8640 10 librados: watch_flush enter 2024-03-25T18:25:32.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.086+0000 7f85199a8640 10 librados: watch_flush exit 2024-03-25T18:25:32.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.086+0000 7f85199a8640 20 client.4203.objecter shutdown clearing up homeless session... 2024-03-25T18:25:32.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.086+0000 7f85199a8640 10 client.4203.objecter successfully canceled tick 2024-03-25T18:25:32.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.086+0000 7f85199a8640 1 -- 172.21.15.99:0/2996302852 >> v1:172.21.15.190:6800/4220409699 conn(0x7f850006e0a0 legacy=0x7f8500070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.086+0000 7f85199a8640 10 monclient: shutdown 2024-03-25T18:25:32.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.086+0000 7f85199a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:32.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.086+0000 7f85199a8640 1 -- 172.21.15.99:0/2996302852 >> v1:172.21.15.99:6789/0 conn(0x7f851413f710 legacy=0x7f8514134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.086+0000 7f85199a8640 1 -- 172.21.15.99:0/2996302852 shutdown_connections 2024-03-25T18:25:32.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.086+0000 7f85199a8640 1 -- 172.21.15.99:0/2996302852 >> 172.21.15.99:0/2996302852 conn(0x7f85140b7ea0 msgr2=0x7f8514074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:32.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.086+0000 7f85199a8640 1 -- 172.21.15.99:0/2996302852 shutdown_connections 2024-03-25T18:25:32.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.086+0000 7f85199a8640 1 -- 172.21.15.99:0/2996302852 wait complete. 2024-03-25T18:25:32.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:32.087+0000 7f85199a8640 1 librados: shutdown 2024-03-25T18:25:32.097 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:32.116 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:10.120 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.153 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:10.120 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.156 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:10.123 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.177 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:10.167 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.210 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:10.201 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.226 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:10.185 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.347 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:10.327 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.347 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:10.315 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.404 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:10.408 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.444 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:10.434 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.450 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:10.417 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.481 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:10.472 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.527 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:10.485 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.624 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:10.591 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.635 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:10.615 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.682 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:10.650 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.691 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:10.695 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.716 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:10.706 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.890 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:10.858 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.927 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:10.907 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.945 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:10.913 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.955 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:10.946 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.984 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:10.952 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:32.987 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:10.977 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:33.018 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:11.008 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:33.039 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:11.006 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:33.070 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:11.028 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:33.124 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:11.083 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:33.211 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:11.215 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:33.266 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:11.270 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:33.487 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:11.477 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:33.497 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:11.477 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:33.543 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:11.533 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:33.581 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:11.561 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:38.091 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:16.058 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:38.098 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:16.065 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:38.353 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:16.321 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:38.542 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:16.501 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:38.903 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:39.026 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:17.017 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:39.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.040+0000 7f46654ba640 10 monclient: get_monmap_and_config 2024-03-25T18:25:39.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.040+0000 7f46654ba640 10 monclient: build_initial_monmap 2024-03-25T18:25:39.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.040+0000 7f46654ba640 10 monclient: monmap: 2024-03-25T18:25:39.041 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:25:39.041 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:39.041 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:25:39.041478+0000 2024-03-25T18:25:39.041 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:25:39.041478+0000 2024-03-25T18:25:39.041 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:39.042 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:39.042 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:39.042 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:25:39.042 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:25:39.042 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:39.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.041+0000 7f46654ba640 1 Processor -- start 2024-03-25T18:25:39.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.041+0000 7f46654ba640 1 -- start start 2024-03-25T18:25:39.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.041+0000 7f46654ba640 10 monclient: init 2024-03-25T18:25:39.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.042+0000 7f46654ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:39.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.042+0000 7f46654ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:39.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.042+0000 7f46654ba640 10 monclient(hunting): picked mon.noname-b con 0x7f46600a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:39.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.042+0000 7f46654ba640 10 monclient(hunting): picked mon.noname-c con 0x7f466013f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:39.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.042+0000 7f46654ba640 10 monclient(hunting): picked mon.noname-a con 0x7f46601430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:39.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.042+0000 7f46654ba640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4660142520 con 0x7f46601430c0 2024-03-25T18:25:39.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.042+0000 7f46654ba640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46600b28a0 con 0x7f466013f710 2024-03-25T18:25:39.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.042+0000 7f46654ba640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46600b2a30 con 0x7f46600a98f0 2024-03-25T18:25:39.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.042+0000 7f46654ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:39.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.042+0000 7f46654ba640 10 monclient(hunting): authenticate will time out at 7357.210393s 2024-03-25T18:25:39.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.042+0000 7f465f7fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f46601430c0 0x7f466014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:52588/0 (socket says 172.21.15.99:52588) 2024-03-25T18:25:39.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.042+0000 7f465f7fe640 1 -- 172.21.15.99:0/354187389 learned_addr learned my addr 172.21.15.99:0/354187389 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:25:39.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 1 -- 172.21.15.99:0/354187389 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121736892 0 0) 0x7f46600b2a30 con 0x7f46600a98f0 2024-03-25T18:25:39.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:25:39.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 1 -- 172.21.15.99:0/354187389 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4648003520 con 0x7f46600a98f0 2024-03-25T18:25:39.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 1 -- 172.21.15.99:0/354187389 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656427778 0 0) 0x7f4660142520 con 0x7f46601430c0 2024-03-25T18:25:39.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:25:39.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 1 -- 172.21.15.99:0/354187389 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46600b2a30 con 0x7f46601430c0 2024-03-25T18:25:39.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 1 -- 172.21.15.99:0/354187389 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5399928 0 0) 0x7f46600b28a0 con 0x7f466013f710 2024-03-25T18:25:39.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:25:39.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.046+0000 7f465dffb640 1 -- 172.21.15.99:0/354187389 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4660142520 con 0x7f466013f710 2024-03-25T18:25:39.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 1 -- 172.21.15.99:0/354187389 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 348398417 0 0) 0x7f46600b2a30 con 0x7f46601430c0 2024-03-25T18:25:39.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 1 -- 172.21.15.99:0/354187389 >> v1:172.21.15.99:6790/0 conn(0x7f46600a98f0 legacy=0x7f46600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 1 -- 172.21.15.99:0/354187389 >> v1:172.21.15.190:6789/0 conn(0x7f466013f710 legacy=0x7f46601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:39.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 1 monclient: found mon.noname-a 2024-03-25T18:25:39.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:39.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:25:39.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 1 -- 172.21.15.99:0/354187389 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46600b44c0 con 0x7f46601430c0 2024-03-25T18:25:39.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:39.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:39.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:39.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 1 -- 172.21.15.99:0/354187389 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f46580019f0 con 0x7f46601430c0 2024-03-25T18:25:39.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:39.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f465dffb640 10 monclient: dump: 2024-03-25T18:25:39.052 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:39.052 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:39.052 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:39.052 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:39.052 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.052 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:39.052 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:39.052 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:39.052 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:39.052 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:39.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f46654ba640 5 monclient: authenticate success, global_id 4206 2024-03-25T18:25:39.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.049+0000 7f46654ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:39.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.051+0000 7f465dffb640 1 -- 172.21.15.99:0/354187389 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4658001b50 con 0x7f46601430c0 2024-03-25T18:25:39.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.051+0000 7f465dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:39.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.051+0000 7f46654ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:39.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.051+0000 7f46654ba640 4 set_mon_vals no callback set 2024-03-25T18:25:39.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.051+0000 7f465dffb640 1 -- 172.21.15.99:0/354187389 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4658001f60 con 0x7f46601430c0 2024-03-25T18:25:39.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.051+0000 7f465dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.051+0000 7f465dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:39.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.051+0000 7f465dffb640 10 monclient: dump: 2024-03-25T18:25:39.054 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:39.054 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:39.054 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:39.054 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:39.054 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.054 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:39.054 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:39.054 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:39.054 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:39.055 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:39.055 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.052+0000 7f46654ba640 10 monclient: shutdown 2024-03-25T18:25:39.055 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.052+0000 7f46654ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:39.055 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.052+0000 7f46654ba640 1 -- 172.21.15.99:0/354187389 >> v1:172.21.15.99:6789/0 conn(0x7f46601430c0 legacy=0x7f466014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.055 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.052+0000 7f46654ba640 1 -- 172.21.15.99:0/354187389 shutdown_connections 2024-03-25T18:25:39.055 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.052+0000 7f46654ba640 1 -- 172.21.15.99:0/354187389 >> 172.21.15.99:0/354187389 conn(0x7f46600b7ea0 msgr2=0x7f46600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:39.055 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.052+0000 7f46654ba640 1 -- 172.21.15.99:0/354187389 shutdown_connections 2024-03-25T18:25:39.055 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.052+0000 7f46654ba640 1 -- 172.21.15.99:0/354187389 wait complete. 2024-03-25T18:25:39.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.053+0000 7f46654ba640 10 monclient: build_initial_monmap 2024-03-25T18:25:39.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.053+0000 7f46654ba640 10 monclient: monmap: 2024-03-25T18:25:39.057 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:25:39.057 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:39.057 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:25:39.057 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:25:39.057 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:39.057 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:25:39.057 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:39.057 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:39.057 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:39.057 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:39.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.054+0000 7f46654ba640 1 librados: starting msgr at 2024-03-25T18:25:39.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.054+0000 7f46654ba640 1 librados: starting objecter 2024-03-25T18:25:39.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.054+0000 7f46654ba640 1 Processor -- start 2024-03-25T18:25:39.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.054+0000 7f46654ba640 1 -- start start 2024-03-25T18:25:39.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.054+0000 7f46654ba640 1 librados: setting wanted keys 2024-03-25T18:25:39.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.055+0000 7f46654ba640 1 librados: calling monclient init 2024-03-25T18:25:39.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.055+0000 7f46654ba640 10 monclient: init 2024-03-25T18:25:39.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.056+0000 7f46654ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:39.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.056+0000 7f46654ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:39.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.056+0000 7f46654ba640 10 monclient(hunting): picked mon.noname-a con 0x7f46600a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:39.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.056+0000 7f46654ba640 10 monclient(hunting): picked mon.noname-c con 0x7f466013f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:39.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.056+0000 7f46654ba640 10 monclient(hunting): picked mon.noname-b con 0x7f46601430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:39.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.056+0000 7f46654ba640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46601da8f0 con 0x7f46600a98f0 2024-03-25T18:25:39.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.056+0000 7f465effd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f46600a98f0 0x7f46601db060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:52592/0 (socket says 172.21.15.99:52592) 2024-03-25T18:25:39.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.056+0000 7f465effd640 1 -- 172.21.15.99:0/3016647502 learned_addr learned my addr 172.21.15.99:0/3016647502 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:25:39.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.056+0000 7f46654ba640 1 -- 172.21.15.99:0/3016647502 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46601db770 con 0x7f46601430c0 2024-03-25T18:25:39.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.057+0000 7f46654ba640 1 -- 172.21.15.99:0/3016647502 --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46601dab40 con 0x7f466013f710 2024-03-25T18:25:39.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.057+0000 7f46654ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:39.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.057+0000 7f46477fe640 10 client.?.objecter ms_handle_connect 0x7f46600a98f0 2024-03-25T18:25:39.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.057+0000 7f46477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:39.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.057+0000 7f46654ba640 10 monclient(hunting): authenticate will time out at 7357.225284s 2024-03-25T18:25:39.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.057+0000 7f46477fe640 10 client.?.objecter ms_handle_connect 0x7f46601430c0 2024-03-25T18:25:39.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.057+0000 7f46477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:39.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.057+0000 7f46477fe640 10 client.?.objecter ms_handle_connect 0x7f466013f710 2024-03-25T18:25:39.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.057+0000 7f46477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:39.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.060+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766126863 0 0) 0x7f46601db770 con 0x7f46601430c0 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.060+0000 7f46477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.060+0000 7f46477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.060+0000 7f46477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.060+0000 7f46477fe640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.060+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f464c004eb0 con 0x7f46601430c0 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.060+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467697118 0 0) 0x7f46601da8f0 con 0x7f46600a98f0 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.060+0000 7f46477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.060+0000 7f46477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.060+0000 7f46477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.060+0000 7f46477fe640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.060+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46601db770 con 0x7f46600a98f0 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.061+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339279334 0 0) 0x7f46601dab40 con 0x7f466013f710 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.061+0000 7f46477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.061+0000 7f46477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.061+0000 7f46477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.061+0000 7f46477fe640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:25:39.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.061+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46601da8f0 con 0x7f466013f710 2024-03-25T18:25:39.065 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.064+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2830009750 0 0) 0x7f46601db770 con 0x7f46600a98f0 2024-03-25T18:25:39.065 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.064+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46601dab40 con 0x7f46600a98f0 2024-03-25T18:25:39.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.065+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f46500019b0 con 0x7f46600a98f0 2024-03-25T18:25:39.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.065+0000 7f46477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.065+0000 7f46477fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:39.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.065+0000 7f46477fe640 10 monclient(hunting): dump: 2024-03-25T18:25:39.067 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:39.067 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:39.067 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:39.067 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:39.067 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.067 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:39.067 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:39.068 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:39.068 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:39.068 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:39.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.065+0000 7f46477fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:39.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.066+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3589148370 0 0) 0x7f464c004eb0 con 0x7f46601430c0 2024-03-25T18:25:39.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.066+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f464c005ad0 con 0x7f46601430c0 2024-03-25T18:25:39.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.066+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4658001ea0 con 0x7f46601430c0 2024-03-25T18:25:39.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.066+0000 7f46477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.066+0000 7f46477fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:39.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.066+0000 7f46477fe640 10 monclient(hunting): dump: 2024-03-25T18:25:39.068 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:39.068 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:39.068 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:39.068 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:39.069 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.069 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:39.069 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:39.069 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:39.069 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:39.069 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:39.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.068+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 892061661 0 0) 0x7f46601da8f0 con 0x7f466013f710 2024-03-25T18:25:39.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.068+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f464c004eb0 con 0x7f466013f710 2024-03-25T18:25:39.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.068+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f46540017e0 con 0x7f466013f710 2024-03-25T18:25:39.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.068+0000 7f46477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.068+0000 7f46477fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:39.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.068+0000 7f46477fe640 10 monclient(hunting): dump: 2024-03-25T18:25:39.070 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:39.070 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:39.070 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:39.070 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:39.070 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.070 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:39.070 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:39.070 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:39.070 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:39.070 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:39.073 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2665866758 0 0) 0x7f46601dab40 con 0x7f46600a98f0 2024-03-25T18:25:39.073 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 >> v1:172.21.15.99:6790/0 conn(0x7f466013f710 legacy=0x7f4660138420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.073 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 >> v1:172.21.15.190:6789/0 conn(0x7f46601430c0 legacy=0x7f46601e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.073 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46477fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:39.073 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46477fe640 1 monclient: found mon.a 2024-03-25T18:25:39.073 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:39.073 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46477fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:39.073 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4660138b30 con 0x7f46600a98f0 2024-03-25T18:25:39.073 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46477fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:39.073 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46477fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:39.073 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:39.073 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46654ba640 5 monclient: authenticate success, global_id 4209 2024-03-25T18:25:39.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46654ba640 10 monclient: _renew_subs 2024-03-25T18:25:39.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46654ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:39.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46654ba640 1 -- 172.21.15.99:0/3016647502 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4660137d10 con 0x7f46600a98f0 2024-03-25T18:25:39.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46654ba640 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:39.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46654ba640 10 monclient: _renew_subs 2024-03-25T18:25:39.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46654ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:39.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46654ba640 1 -- 172.21.15.99:0/3016647502 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46601e3520 con 0x7f46600a98f0 2024-03-25T18:25:39.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.069+0000 7f46654ba640 1 librados: init done 2024-03-25T18:25:39.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.071+0000 7f46654ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:39.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.071+0000 7f46654ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:39.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.071+0000 7f46654ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:39.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.071+0000 7f46654ba640 1 -- 172.21.15.99:0/3016647502 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46600b3ef0 con 0x7f46600a98f0 2024-03-25T18:25:39.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.094+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46500035f0 con 0x7f46600a98f0 2024-03-25T18:25:39.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.094+0000 7f46477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:39.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.094+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f46500019b0 con 0x7f46600a98f0 2024-03-25T18:25:39.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.094+0000 7f46477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.094+0000 7f46477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:39.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.094+0000 7f46477fe640 10 monclient: dump: 2024-03-25T18:25:39.096 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:39.097 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:39.097 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:39.097 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:39.097 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.097 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:39.097 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:39.097 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:39.097 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:39.097 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:39.097 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.094+0000 7f465d7fa640 4 set_mon_vals no callback set 2024-03-25T18:25:39.098 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.095+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f46500038a0 con 0x7f46600a98f0 2024-03-25T18:25:39.098 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.096+0000 7f465e7fc640 10 monclient: get_auth_request con 0x7f464c06e4f0 auth_method 0 2024-03-25T18:25:39.098 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.096+0000 7f46477fe640 10 client.4209.objecter ms_handle_connect 0x7f464c06e4f0 2024-03-25T18:25:39.101 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.100+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f4650054420 con 0x7f46600a98f0 2024-03-25T18:25:39.101 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.100+0000 7f46477fe640 10 client.4209.objecter ms_dispatch 0x7f4660000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:39.101 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.100+0000 7f46477fe640 3 client.4209.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:39.101 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.100+0000 7f46477fe640 3 client.4209.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:39.101 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.100+0000 7f46477fe640 20 client.4209.objecter dump_active .. 0 homeless 2024-03-25T18:25:39.329 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:17.333 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:39.339 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:17.330 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:39.417 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.416+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f465004ff30 con 0x7f46600a98f0 2024-03-25T18:25:39.417 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.416+0000 7f46477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:39.417 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.416+0000 7f46477fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:39.510 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:17.491 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:39.744 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.743+0000 7f46654ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:39.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.743+0000 7f46654ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:39.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.743+0000 7f46654ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:25:39.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.743+0000 7f46654ba640 1 -- 172.21.15.99:0/3016647502 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46601366b0 con 0x7f46600a98f0 2024-03-25T18:25:39.769 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.768+0000 7f46477fe640 1 -- 172.21.15.99:0/3016647502 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f4650051e60 con 0x7f46600a98f0 2024-03-25T18:25:39.769 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.768+0000 7f46477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:39.769 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.768+0000 7f46477fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:39.769 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:25:39.769 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:39.771 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.770+0000 7f46654ba640 10 librados: watch_flush enter 2024-03-25T18:25:39.771 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.770+0000 7f46654ba640 10 librados: watch_flush exit 2024-03-25T18:25:39.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.770+0000 7f46654ba640 20 client.4209.objecter shutdown clearing up homeless session... 2024-03-25T18:25:39.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.770+0000 7f46654ba640 10 client.4209.objecter successfully canceled tick 2024-03-25T18:25:39.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.770+0000 7f46654ba640 1 -- 172.21.15.99:0/3016647502 >> v1:172.21.15.190:6800/4220409699 conn(0x7f464c06e4f0 legacy=0x7f464c0709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.770+0000 7f46654ba640 10 monclient: shutdown 2024-03-25T18:25:39.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.770+0000 7f46654ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:39.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.770+0000 7f46654ba640 1 -- 172.21.15.99:0/3016647502 >> v1:172.21.15.99:6789/0 conn(0x7f46600a98f0 legacy=0x7f46601db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.770+0000 7f46654ba640 1 -- 172.21.15.99:0/3016647502 shutdown_connections 2024-03-25T18:25:39.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.770+0000 7f46654ba640 1 -- 172.21.15.99:0/3016647502 >> 172.21.15.99:0/3016647502 conn(0x7f46600b7ea0 msgr2=0x7f4660074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:39.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.771+0000 7f46654ba640 1 -- 172.21.15.99:0/3016647502 shutdown_connections 2024-03-25T18:25:39.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.771+0000 7f46654ba640 1 -- 172.21.15.99:0/3016647502 wait complete. 2024-03-25T18:25:39.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:39.771+0000 7f46654ba640 1 librados: shutdown 2024-03-25T18:25:39.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:43.360 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:21.327 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:43.364 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:21.331 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:43.568 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:21.535 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:43.572 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:21.539 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:43.812 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:21.780 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:43.818 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:21.786 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:44.473 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:22.432 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:44.477 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:22.436 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:44.480 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:22.471 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:44.484 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:22.475 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:44.934 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:22.924 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:44.938 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:22.928 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:45.027 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:23.032 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:45.031 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:23.036 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:45.132 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:23.112 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:45.137 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:23.118 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:46.591 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:46.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.678+0000 7f5f6cc50640 10 monclient: get_monmap_and_config 2024-03-25T18:25:46.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.678+0000 7f5f6cc50640 10 monclient: build_initial_monmap 2024-03-25T18:25:46.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.678+0000 7f5f6cc50640 10 monclient: monmap: 2024-03-25T18:25:46.679 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:25:46.679 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:46.679 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:25:46.679512+0000 2024-03-25T18:25:46.680 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:25:46.679512+0000 2024-03-25T18:25:46.680 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:46.680 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:46.680 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:46.680 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:25:46.680 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:25:46.680 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:46.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.679+0000 7f5f6cc50640 1 Processor -- start 2024-03-25T18:25:46.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.679+0000 7f5f6cc50640 1 -- start start 2024-03-25T18:25:46.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.679+0000 7f5f6cc50640 10 monclient: init 2024-03-25T18:25:46.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.679+0000 7f5f6cc50640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:46.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.679+0000 7f5f6cc50640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:46.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.679+0000 7f5f6cc50640 10 monclient(hunting): picked mon.noname-b con 0x7f5f680b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:46.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.679+0000 7f5f6cc50640 10 monclient(hunting): picked mon.noname-c con 0x7f5f680b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:46.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.679+0000 7f5f6cc50640 10 monclient(hunting): picked mon.noname-a con 0x7f5f6814be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:46.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.679+0000 7f5f6cc50640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f680af940 con 0x7f5f6814be90 2024-03-25T18:25:46.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.679+0000 7f5f6cc50640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f680b0a70 con 0x7f5f680b1730 2024-03-25T18:25:46.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.680+0000 7f5f6cc50640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f680b0d50 con 0x7f5f680b4d60 2024-03-25T18:25:46.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.680+0000 7f5f6cc50640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:46.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.680+0000 7f5f6cc50640 10 monclient(hunting): authenticate will time out at 7364.847909s 2024-03-25T18:25:46.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.680+0000 7f5f6659c640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f5f680b4d60 0x7f5f6813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:46766/0 (socket says 172.21.15.99:46766) 2024-03-25T18:25:46.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.680+0000 7f5f6659c640 1 -- 172.21.15.99:0/2634025226 learned_addr learned my addr 172.21.15.99:0/2634025226 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:25:46.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.683+0000 7f5f6559a640 1 -- 172.21.15.99:0/2634025226 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206649170 0 0) 0x7f5f680af940 con 0x7f5f6814be90 2024-03-25T18:25:46.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.683+0000 7f5f6559a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:46.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.683+0000 7f5f6559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:46.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.683+0000 7f5f6559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:46.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.683+0000 7f5f6559a640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:25:46.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.683+0000 7f5f6559a640 1 -- 172.21.15.99:0/2634025226 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f50003520 con 0x7f5f6814be90 2024-03-25T18:25:46.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.683+0000 7f5f6559a640 1 -- 172.21.15.99:0/2634025226 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242427457 0 0) 0x7f5f680b0d50 con 0x7f5f680b4d60 2024-03-25T18:25:46.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.683+0000 7f5f6559a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:46.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.683+0000 7f5f6559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:46.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.683+0000 7f5f6559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:46.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.683+0000 7f5f6559a640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:25:46.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.683+0000 7f5f6559a640 1 -- 172.21.15.99:0/2634025226 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f680af940 con 0x7f5f680b4d60 2024-03-25T18:25:46.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.684+0000 7f5f6559a640 1 -- 172.21.15.99:0/2634025226 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484082404 0 0) 0x7f5f680b0a70 con 0x7f5f680b1730 2024-03-25T18:25:46.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.684+0000 7f5f6559a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:46.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.684+0000 7f5f6559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:46.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.684+0000 7f5f6559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:46.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.684+0000 7f5f6559a640 10 monclient(hunting): my global_id is 4225 2024-03-25T18:25:46.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.684+0000 7f5f6559a640 1 -- 172.21.15.99:0/2634025226 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f680b0d50 con 0x7f5f680b1730 2024-03-25T18:25:46.687 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.686+0000 7f5f6559a640 1 -- 172.21.15.99:0/2634025226 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1066814600 0 0) 0x7f5f680af940 con 0x7f5f680b4d60 2024-03-25T18:25:46.687 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.686+0000 7f5f6559a640 1 -- 172.21.15.99:0/2634025226 >> v1:172.21.15.190:6789/0 conn(0x7f5f680b1730 legacy=0x7f5f680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:46.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.686+0000 7f5f6559a640 1 -- 172.21.15.99:0/2634025226 >> v1:172.21.15.99:6789/0 conn(0x7f5f6814be90 legacy=0x7f5f6814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:46.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.686+0000 7f5f6559a640 10 monclient: _finish_hunting 0 2024-03-25T18:25:46.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.686+0000 7f5f6559a640 1 monclient: found mon.noname-b 2024-03-25T18:25:46.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.686+0000 7f5f6559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:46.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.686+0000 7f5f6559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:25:46.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.686+0000 7f5f6559a640 1 -- 172.21.15.99:0/2634025226 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f680b1030 con 0x7f5f680b4d60 2024-03-25T18:25:46.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.686+0000 7f5f6559a640 10 monclient: _finish_auth 0 2024-03-25T18:25:46.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.686+0000 7f5f6559a640 10 monclient: _check_auth_tickets 2024-03-25T18:25:46.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.686+0000 7f5f6559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:46.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.686+0000 7f5f6cc50640 5 monclient: authenticate success, global_id 4220 2024-03-25T18:25:46.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.686+0000 7f5f6cc50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:46.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.688+0000 7f5f6559a640 1 -- 172.21.15.99:0/2634025226 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5f58001560 con 0x7f5f680b4d60 2024-03-25T18:25:46.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.688+0000 7f5f6559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:46.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.688+0000 7f5f6559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:46.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.688+0000 7f5f6559a640 10 monclient: dump: 2024-03-25T18:25:46.690 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:46.690 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:46.691 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:46.691 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:46.691 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:46.691 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:46.691 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:46.691 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:46.691 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:46.692 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:46.692 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.688+0000 7f5f6cc50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:46.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.691+0000 7f5f6559a640 1 -- 172.21.15.99:0/2634025226 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f580017b0 con 0x7f5f680b4d60 2024-03-25T18:25:46.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.691+0000 7f5f6559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:46.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.691+0000 7f5f6cc50640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:46.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.691+0000 7f5f6cc50640 4 set_mon_vals no callback set 2024-03-25T18:25:46.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.692+0000 7f5f6cc50640 10 monclient: shutdown 2024-03-25T18:25:46.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.692+0000 7f5f6cc50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:46.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.692+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2634025226 >> v1:172.21.15.99:6790/0 conn(0x7f5f680b4d60 legacy=0x7f5f6813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:46.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.693+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2634025226 shutdown_connections 2024-03-25T18:25:46.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.693+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2634025226 >> 172.21.15.99:0/2634025226 conn(0x7f5f680617a0 msgr2=0x7f5f68061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:46.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.693+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2634025226 shutdown_connections 2024-03-25T18:25:46.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.693+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2634025226 wait complete. 2024-03-25T18:25:46.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.694+0000 7f5f6cc50640 10 monclient: build_initial_monmap 2024-03-25T18:25:46.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.694+0000 7f5f6cc50640 10 monclient: monmap: 2024-03-25T18:25:46.699 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:25:46.699 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:46.699 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:25:46.699 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:25:46.699 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:46.699 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:25:46.699 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:46.699 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.695+0000 7f5f6cc50640 1 librados: starting msgr at 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.695+0000 7f5f6cc50640 1 librados: starting objecter 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.695+0000 7f5f6cc50640 1 Processor -- start 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.695+0000 7f5f6cc50640 1 -- start start 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.695+0000 7f5f6cc50640 1 librados: setting wanted keys 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.695+0000 7f5f6cc50640 1 librados: calling monclient init 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.695+0000 7f5f6cc50640 10 monclient: init 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.696+0000 7f5f6cc50640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.696+0000 7f5f6cc50640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.697+0000 7f5f6cc50640 10 monclient(hunting): picked mon.noname-a con 0x7f5f680b1730 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.697+0000 7f5f6cc50640 10 monclient(hunting): picked mon.noname-c con 0x7f5f680b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.697+0000 7f5f6cc50640 10 monclient(hunting): picked mon.noname-b con 0x7f5f6814be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.697+0000 7f5f6cc50640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f681431c0 con 0x7f5f680b1730 2024-03-25T18:25:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.697+0000 7f5f6cc50640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6813b0e0 con 0x7f5f6814be90 2024-03-25T18:25:46.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.697+0000 7f5f6cc50640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6813b270 con 0x7f5f680b4d60 2024-03-25T18:25:46.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.697+0000 7f5f6cc50640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:46.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.697+0000 7f5f6cc50640 10 monclient(hunting): authenticate will time out at 7364.865029s 2024-03-25T18:25:46.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.697+0000 7f5f6659c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f5f680b1730 0x7f5f681425e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53114/0 (socket says 172.21.15.99:53114) 2024-03-25T18:25:46.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.697+0000 7f5f6659c640 1 -- 172.21.15.99:0/2732324967 learned_addr learned my addr 172.21.15.99:0/2732324967 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:25:46.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.697+0000 7f5f56ffd640 10 client.?.objecter ms_handle_connect 0x7f5f680b1730 2024-03-25T18:25:46.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.697+0000 7f5f56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:46.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.698+0000 7f5f56ffd640 10 client.?.objecter ms_handle_connect 0x7f5f6814be90 2024-03-25T18:25:46.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.698+0000 7f5f56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:46.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.699+0000 7f5f56ffd640 10 client.?.objecter ms_handle_connect 0x7f5f680b4d60 2024-03-25T18:25:46.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.699+0000 7f5f56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:46.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.747+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408669971 0 0) 0x7f5f6813b270 con 0x7f5f680b4d60 2024-03-25T18:25:46.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.747+0000 7f5f56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:46.749 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.747+0000 7f5f56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:46.749 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.747+0000 7f5f56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:46.749 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.747+0000 7f5f56ffd640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:25:46.749 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.747+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f48004eb0 con 0x7f5f680b4d60 2024-03-25T18:25:46.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.752+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1289966513 0 0) 0x7f5f48004eb0 con 0x7f5f680b4d60 2024-03-25T18:25:46.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.752+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f6813b270 con 0x7f5f680b4d60 2024-03-25T18:25:46.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.752+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5f5c0018f0 con 0x7f5f680b4d60 2024-03-25T18:25:46.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.752+0000 7f5f56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:46.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.752+0000 7f5f56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:46.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.752+0000 7f5f56ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:46.754 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:46.754 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:46.754 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:46.754 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:46.754 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:46.754 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:46.754 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:46.754 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:46.754 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:46.754 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:46.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.752+0000 7f5f56ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:46.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 179556162 0 0) 0x7f5f6813b270 con 0x7f5f680b4d60 2024-03-25T18:25:46.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 >> v1:172.21.15.190:6789/0 conn(0x7f5f6814be90 legacy=0x7f5f681d8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:46.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 >> v1:172.21.15.99:6789/0 conn(0x7f5f680b1730 legacy=0x7f5f681425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:46.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f56ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:46.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f56ffd640 1 monclient: found mon.c 2024-03-25T18:25:46.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:46.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f56ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:46.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6813b400 con 0x7f5f680b4d60 2024-03-25T18:25:46.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f56ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:46.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f56ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:46.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:46.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f6cc50640 5 monclient: authenticate success, global_id 4223 2024-03-25T18:25:46.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f6cc50640 10 monclient: _renew_subs 2024-03-25T18:25:46.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f6cc50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:46.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2732324967 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f681e3380 con 0x7f5f680b4d60 2024-03-25T18:25:46.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f6cc50640 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:46.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f6cc50640 10 monclient: _renew_subs 2024-03-25T18:25:46.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f6cc50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:46.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2732324967 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f681e36f0 con 0x7f5f680b4d60 2024-03-25T18:25:46.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.756+0000 7f5f6cc50640 1 librados: init done 2024-03-25T18:25:46.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.757+0000 7f5f6cc50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:46.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.757+0000 7f5f6cc50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:46.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.757+0000 7f5f6cc50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:46.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.757+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2732324967 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f681401f0 con 0x7f5f680b4d60 2024-03-25T18:25:46.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.782+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f5c003610 con 0x7f5f680b4d60 2024-03-25T18:25:46.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.782+0000 7f5f56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:46.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.782+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5f5c0039e0 con 0x7f5f680b4d60 2024-03-25T18:25:46.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.782+0000 7f5f56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:46.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.782+0000 7f5f56ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:46.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.782+0000 7f5f56ffd640 10 monclient: dump: 2024-03-25T18:25:46.783 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:46.783 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:46.784 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:46.784 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:46.784 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:46.784 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:46.784 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:46.784 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:46.784 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:46.784 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:46.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.782+0000 7f5f64d99640 4 set_mon_vals no callback set 2024-03-25T18:25:46.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.782+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f5f5c003c00 con 0x7f5f680b4d60 2024-03-25T18:25:46.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.783+0000 7f5f6659c640 10 monclient: get_auth_request con 0x7f5f4806ddc0 auth_method 0 2024-03-25T18:25:46.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:46.783+0000 7f5f56ffd640 10 client.4223.objecter ms_handle_connect 0x7f5f4806ddc0 2024-03-25T18:25:47.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.059+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f5f5c0537a0 con 0x7f5f680b4d60 2024-03-25T18:25:47.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.059+0000 7f5f56ffd640 10 client.4223.objecter ms_dispatch 0x7f5f68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:47.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.059+0000 7f5f56ffd640 3 client.4223.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:47.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.059+0000 7f5f56ffd640 3 client.4223.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:47.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.059+0000 7f5f56ffd640 20 client.4223.objecter dump_active .. 0 homeless 2024-03-25T18:25:47.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.059+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f5c04f2b0 con 0x7f5f680b4d60 2024-03-25T18:25:47.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.059+0000 7f5f56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:47.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.059+0000 7f5f56ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:47.242 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.241+0000 7f5f6cc50640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:47.242 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.241+0000 7f5f6cc50640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:47.242 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.241+0000 7f5f6cc50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:47.242 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.241+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2732324967 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f680627b0 con 0x7f5f680b4d60 2024-03-25T18:25:47.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.257+0000 7f5f56ffd640 1 -- 172.21.15.99:0/2732324967 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f5f5c0511e0 con 0x7f5f680b4d60 2024-03-25T18:25:47.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.257+0000 7f5f56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:47.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.257+0000 7f5f56ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:47.258 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:25:47.258 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:47.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.259+0000 7f5f6cc50640 10 librados: watch_flush enter 2024-03-25T18:25:47.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.259+0000 7f5f6cc50640 10 librados: watch_flush exit 2024-03-25T18:25:47.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.259+0000 7f5f6cc50640 20 client.4223.objecter shutdown clearing up homeless session... 2024-03-25T18:25:47.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.259+0000 7f5f6cc50640 10 client.4223.objecter successfully canceled tick 2024-03-25T18:25:47.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.259+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2732324967 >> v1:172.21.15.190:6800/4220409699 conn(0x7f5f4806ddc0 legacy=0x7f5f48070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.259+0000 7f5f6cc50640 10 monclient: shutdown 2024-03-25T18:25:47.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.259+0000 7f5f6cc50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:47.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.259+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2732324967 >> v1:172.21.15.99:6790/0 conn(0x7f5f680b4d60 legacy=0x7f5f681d7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.259+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2732324967 shutdown_connections 2024-03-25T18:25:47.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.259+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2732324967 >> 172.21.15.99:0/2732324967 conn(0x7f5f680617a0 msgr2=0x7f5f6813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:47.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.260+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2732324967 shutdown_connections 2024-03-25T18:25:47.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.260+0000 7f5f6cc50640 1 -- 172.21.15.99:0/2732324967 wait complete. 2024-03-25T18:25:47.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:47.260+0000 7f5f6cc50640 1 librados: shutdown 2024-03-25T18:25:47.271 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:47.597 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:00:25.564 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:47.877 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:00:25.844 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:47.901 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:00:25.869 35305== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:48.566 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:00:26.557 35544== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:48.637 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:00:26.596 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:48.985 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:00:26.975 35545== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:49.452 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:00:27.457 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:49.633 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:00:27.613 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T18:25:54.078 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:54.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.168+0000 7f47f7b3d640 10 monclient: get_monmap_and_config 2024-03-25T18:25:54.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.168+0000 7f47f7b3d640 10 monclient: build_initial_monmap 2024-03-25T18:25:54.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.168+0000 7f47f7b3d640 10 monclient: monmap: 2024-03-25T18:25:54.173 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:25:54.173 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:54.173 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:25:54.169657+0000 2024-03-25T18:25:54.173 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:25:54.169657+0000 2024-03-25T18:25:54.173 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:54.173 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:54.174 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:54.174 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:25:54.174 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:25:54.174 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:54.174 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.169+0000 7f47f7b3d640 1 Processor -- start 2024-03-25T18:25:54.174 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.169+0000 7f47f7b3d640 1 -- start start 2024-03-25T18:25:54.175 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.169+0000 7f47f7b3d640 10 monclient: init 2024-03-25T18:25:54.175 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.169+0000 7f47f7b3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:54.175 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.169+0000 7f47f7b3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:54.175 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.170+0000 7f47f7b3d640 10 monclient(hunting): picked mon.noname-c con 0x7f47f00b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:54.175 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.170+0000 7f47f7b3d640 10 monclient(hunting): picked mon.noname-a con 0x7f47f00b1730 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:54.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.170+0000 7f47f7b3d640 10 monclient(hunting): picked mon.noname-b con 0x7f47f014be90 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:54.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.170+0000 7f47f7b3d640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f00af940 con 0x7f47f00b1730 2024-03-25T18:25:54.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.170+0000 7f47f7b3d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f00b0a70 con 0x7f47f00b4d60 2024-03-25T18:25:54.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.170+0000 7f47f7b3d640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f00b0d50 con 0x7f47f014be90 2024-03-25T18:25:54.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.170+0000 7f47f7b3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:54.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.170+0000 7f47f7b3d640 10 monclient(hunting): authenticate will time out at 7372.338146s 2024-03-25T18:25:54.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.170+0000 7f47f50d8640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f47f00b1730 0x7f47f00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53128/0 (socket says 172.21.15.99:53128) 2024-03-25T18:25:54.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.170+0000 7f47f50d8640 1 -- 172.21.15.99:0/428139614 learned_addr learned my addr 172.21.15.99:0/428139614 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:25:54.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.173+0000 7f47f48d7640 1 -- 172.21.15.99:0/428139614 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294101215 0 0) 0x7f47f00af940 con 0x7f47f00b1730 2024-03-25T18:25:54.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.173+0000 7f47f48d7640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.173+0000 7f47f48d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.173+0000 7f47f48d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.173+0000 7f47f48d7640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:25:54.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.173+0000 7f47f48d7640 1 -- 172.21.15.99:0/428139614 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47dc003520 con 0x7f47f00b1730 2024-03-25T18:25:54.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.174+0000 7f47f48d7640 1 -- 172.21.15.99:0/428139614 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521768952 0 0) 0x7f47f00b0d50 con 0x7f47f014be90 2024-03-25T18:25:54.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.174+0000 7f47f48d7640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.174+0000 7f47f48d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.174+0000 7f47f48d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.174+0000 7f47f48d7640 10 monclient(hunting): my global_id is 4226 2024-03-25T18:25:54.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.174+0000 7f47f48d7640 1 -- 172.21.15.99:0/428139614 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f00af940 con 0x7f47f014be90 2024-03-25T18:25:54.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.174+0000 7f47f48d7640 1 -- 172.21.15.99:0/428139614 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101761399 0 0) 0x7f47f00b0a70 con 0x7f47f00b4d60 2024-03-25T18:25:54.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.174+0000 7f47f48d7640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.174+0000 7f47f48d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.174+0000 7f47f48d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.174+0000 7f47f48d7640 10 monclient(hunting): my global_id is 4228 2024-03-25T18:25:54.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.174+0000 7f47f48d7640 1 -- 172.21.15.99:0/428139614 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f00b0d50 con 0x7f47f00b4d60 2024-03-25T18:25:54.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 1 -- 172.21.15.99:0/428139614 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1888837072 0 0) 0x7f47dc003520 con 0x7f47f00b1730 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 1 -- 172.21.15.99:0/428139614 >> v1:172.21.15.99:6790/0 conn(0x7f47f014be90 legacy=0x7f47f014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 1 -- 172.21.15.99:0/428139614 >> v1:172.21.15.190:6789/0 conn(0x7f47f00b4d60 legacy=0x7f47f013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 10 monclient: _finish_hunting 0 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 1 monclient: found mon.noname-a 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 1 -- 172.21.15.99:0/428139614 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f00b1030 con 0x7f47f00b1730 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 10 monclient: _finish_auth 0 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 10 monclient: _check_auth_tickets 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 1 -- 172.21.15.99:0/428139614 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f47e0001690 con 0x7f47f00b1730 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f48d7640 10 monclient: dump: 2024-03-25T18:25:54.185 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:54.186 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:54.186 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:54.186 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:54.186 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:54.186 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:54.186 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:54.186 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:54.186 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:54.186 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:54.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f7b3d640 5 monclient: authenticate success, global_id 4215 2024-03-25T18:25:54.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.177+0000 7f47f7b3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:54.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.179+0000 7f47f48d7640 1 -- 172.21.15.99:0/428139614 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47e00017f0 con 0x7f47f00b1730 2024-03-25T18:25:54.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.179+0000 7f47f48d7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.179+0000 7f47f48d7640 1 -- 172.21.15.99:0/428139614 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f47e0001c00 con 0x7f47f00b1730 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.179+0000 7f47f48d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.179+0000 7f47f48d7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.179+0000 7f47f48d7640 10 monclient: dump: 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.179+0000 7f47f7b3d640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:54.187 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.179+0000 7f47f7b3d640 4 set_mon_vals no callback set 2024-03-25T18:25:54.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.180+0000 7f47f7b3d640 10 monclient: shutdown 2024-03-25T18:25:54.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.180+0000 7f47f7b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:54.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.180+0000 7f47f7b3d640 1 -- 172.21.15.99:0/428139614 >> v1:172.21.15.99:6789/0 conn(0x7f47f00b1730 legacy=0x7f47f00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.180+0000 7f47f7b3d640 1 -- 172.21.15.99:0/428139614 shutdown_connections 2024-03-25T18:25:54.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.180+0000 7f47f7b3d640 1 -- 172.21.15.99:0/428139614 >> 172.21.15.99:0/428139614 conn(0x7f47f00617a0 msgr2=0x7f47f0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:54.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.180+0000 7f47f7b3d640 1 -- 172.21.15.99:0/428139614 shutdown_connections 2024-03-25T18:25:54.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.180+0000 7f47f7b3d640 1 -- 172.21.15.99:0/428139614 wait complete. 2024-03-25T18:25:54.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.181+0000 7f47f7b3d640 10 monclient: build_initial_monmap 2024-03-25T18:25:54.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.181+0000 7f47f7b3d640 10 monclient: monmap: 2024-03-25T18:25:54.188 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:25:54.188 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:54.188 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:25:54.188 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:25:54.188 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:54.189 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:25:54.189 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:25:54.189 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:25:54.189 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:25:54.189 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:54.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.182+0000 7f47f7b3d640 1 librados: starting msgr at 2024-03-25T18:25:54.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.182+0000 7f47f7b3d640 1 librados: starting objecter 2024-03-25T18:25:54.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.182+0000 7f47f7b3d640 1 Processor -- start 2024-03-25T18:25:54.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.182+0000 7f47f7b3d640 1 -- start start 2024-03-25T18:25:54.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.182+0000 7f47f7b3d640 1 librados: setting wanted keys 2024-03-25T18:25:54.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.182+0000 7f47f7b3d640 1 librados: calling monclient init 2024-03-25T18:25:54.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.182+0000 7f47f7b3d640 10 monclient: init 2024-03-25T18:25:54.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.182+0000 7f47f7b3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:54.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.182+0000 7f47f7b3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:54.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.182+0000 7f47f7b3d640 10 monclient(hunting): picked mon.noname-a con 0x7f47f00b1730 addr v1:172.21.15.99:6789/0 2024-03-25T18:25:54.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.182+0000 7f47f7b3d640 10 monclient(hunting): picked mon.noname-c con 0x7f47f00b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:25:54.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.183+0000 7f47f7b3d640 10 monclient(hunting): picked mon.noname-b con 0x7f47f014be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:25:54.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.183+0000 7f47f7b3d640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f0144d40 con 0x7f47f00b1730 2024-03-25T18:25:54.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.183+0000 7f47f7b3d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f01dbac0 con 0x7f47f014be90 2024-03-25T18:25:54.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.183+0000 7f47f7b3d640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f0135d30 con 0x7f47f00b4d60 2024-03-25T18:25:54.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.183+0000 7f47f7b3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:54.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.183+0000 7f47f7b3d640 10 monclient(hunting): authenticate will time out at 7372.351040s 2024-03-25T18:25:54.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.183+0000 7f47f58d9640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f47f00b1730 0x7f47f01db3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53130/0 (socket says 172.21.15.99:53130) 2024-03-25T18:25:54.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.183+0000 7f47f58d9640 1 -- 172.21.15.99:0/483874974 learned_addr learned my addr 172.21.15.99:0/483874974 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:25:54.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.183+0000 7f47e67fc640 10 client.?.objecter ms_handle_connect 0x7f47f00b1730 2024-03-25T18:25:54.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.183+0000 7f47e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:54.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.184+0000 7f47e67fc640 10 client.?.objecter ms_handle_connect 0x7f47f014be90 2024-03-25T18:25:54.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.184+0000 7f47e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:54.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.184+0000 7f47e67fc640 10 client.?.objecter ms_handle_connect 0x7f47f00b4d60 2024-03-25T18:25:54.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.184+0000 7f47e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:54.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.186+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919504607 0 0) 0x7f47f0135d30 con 0x7f47f00b4d60 2024-03-25T18:25:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.186+0000 7f47e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.186+0000 7f47e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.186+0000 7f47e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.186+0000 7f47e67fc640 10 monclient(hunting): my global_id is 4229 2024-03-25T18:25:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.186+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d4004eb0 con 0x7f47f00b4d60 2024-03-25T18:25:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.186+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350334572 0 0) 0x7f47f01dbac0 con 0x7f47f014be90 2024-03-25T18:25:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.186+0000 7f47e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.187+0000 7f47e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.187+0000 7f47e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.187+0000 7f47e67fc640 10 monclient(hunting): my global_id is 4231 2024-03-25T18:25:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.187+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f0135d30 con 0x7f47f014be90 2024-03-25T18:25:54.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.187+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22980589 0 0) 0x7f47f0144d40 con 0x7f47f00b1730 2024-03-25T18:25:54.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.187+0000 7f47e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.187+0000 7f47e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.187+0000 7f47e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.187+0000 7f47e67fc640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:25:54.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.187+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f01dbac0 con 0x7f47f00b1730 2024-03-25T18:25:54.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.193+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3894345943 0 0) 0x7f47d4004eb0 con 0x7f47f00b4d60 2024-03-25T18:25:54.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.193+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47f0144d40 con 0x7f47f00b4d60 2024-03-25T18:25:54.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.194+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f47e0001b00 con 0x7f47f00b4d60 2024-03-25T18:25:54.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.194+0000 7f47e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:54.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.194+0000 7f47e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:54.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.194+0000 7f47e67fc640 10 monclient(hunting): dump: 2024-03-25T18:25:54.198 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:54.198 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:54.199 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:54.199 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:54.199 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:54.199 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:54.199 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:54.199 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:54.199 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:54.199 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:54.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.194+0000 7f47e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:54.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.194+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1964130260 0 0) 0x7f47f0135d30 con 0x7f47f014be90 2024-03-25T18:25:54.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.194+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47d4004eb0 con 0x7f47f014be90 2024-03-25T18:25:54.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.194+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f47e8001c10 con 0x7f47f014be90 2024-03-25T18:25:54.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.194+0000 7f47e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:54.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.194+0000 7f47e67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:54.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.194+0000 7f47e67fc640 10 monclient(hunting): dump: 2024-03-25T18:25:54.200 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:54.200 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:54.200 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:54.200 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:54.200 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:54.200 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:54.201 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:54.201 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:54.201 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:54.201 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:54.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.194+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3017200844 0 0) 0x7f47f01dbac0 con 0x7f47f00b1730 2024-03-25T18:25:54.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.195+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47f0135d30 con 0x7f47f00b1730 2024-03-25T18:25:54.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.195+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f47ec0019b0 con 0x7f47f00b1730 2024-03-25T18:25:54.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.195+0000 7f47e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:54.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.195+0000 7f47e67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:54.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.195+0000 7f47e67fc640 10 monclient(hunting): dump: 2024-03-25T18:25:54.201 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:54.201 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:54.201 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:54.201 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:54.202 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:54.202 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:54.202 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:54.202 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:54.202 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:54.202 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:54.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4132977962 0 0) 0x7f47f0144d40 con 0x7f47f00b4d60 2024-03-25T18:25:54.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 >> v1:172.21.15.190:6789/0 conn(0x7f47f014be90 legacy=0x7f47f01e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 >> v1:172.21.15.99:6789/0 conn(0x7f47f00b1730 legacy=0x7f47f01db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47e67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:54.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47e67fc640 1 monclient: found mon.c 2024-03-25T18:25:54.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:54.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47e67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:54.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f01382f0 con 0x7f47f00b4d60 2024-03-25T18:25:54.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47e67fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:54.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47e67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:54.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:54.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47f7b3d640 5 monclient: authenticate success, global_id 4229 2024-03-25T18:25:54.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47f7b3d640 10 monclient: _renew_subs 2024-03-25T18:25:54.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47f7b3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:54.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.198+0000 7f47f7b3d640 1 -- 172.21.15.99:0/483874974 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f0145670 con 0x7f47f00b4d60 2024-03-25T18:25:54.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.199+0000 7f47f7b3d640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:54.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.199+0000 7f47f7b3d640 10 monclient: _renew_subs 2024-03-25T18:25:54.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.199+0000 7f47f7b3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:54.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.199+0000 7f47f7b3d640 1 -- 172.21.15.99:0/483874974 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f0145920 con 0x7f47f00b4d60 2024-03-25T18:25:54.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.199+0000 7f47f7b3d640 1 librados: init done 2024-03-25T18:25:54.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.200+0000 7f47f7b3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:54.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.200+0000 7f47f7b3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:54.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.200+0000 7f47f7b3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:54.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.200+0000 7f47f7b3d640 1 -- 172.21.15.99:0/483874974 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f00627b0 con 0x7f47f00b4d60 2024-03-25T18:25:54.477 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.475+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47e0004a90 con 0x7f47f00b4d60 2024-03-25T18:25:54.477 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.475+0000 7f47e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:54.477 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.475+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f47e0001c60 con 0x7f47f00b4d60 2024-03-25T18:25:54.477 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.475+0000 7f47e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:54.477 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.475+0000 7f47e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:54.477 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.475+0000 7f47e67fc640 10 monclient: dump: 2024-03-25T18:25:54.477 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:25:54.477 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:25:54.477 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:54.477 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:25:54.478 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:54.478 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:25:54.478 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:25:54.478 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:25:54.478 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:25:54.478 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:25:54.478 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.475+0000 7f47e7fff640 4 set_mon_vals no callback set 2024-03-25T18:25:54.479 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.477+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f47e001ac10 con 0x7f47f00b4d60 2024-03-25T18:25:54.479 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.477+0000 7f47f58d9640 10 monclient: get_auth_request con 0x7f47d406e0b0 auth_method 0 2024-03-25T18:25:54.479 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.477+0000 7f47e67fc640 10 client.4229.objecter ms_handle_connect 0x7f47d406e0b0 2024-03-25T18:25:54.485 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.484+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f47e00547f0 con 0x7f47f00b4d60 2024-03-25T18:25:54.485 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.484+0000 7f47e67fc640 10 client.4229.objecter ms_dispatch 0x7f47f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:54.485 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.484+0000 7f47e67fc640 3 client.4229.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:54.485 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.484+0000 7f47e67fc640 3 client.4229.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:54.486 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.484+0000 7f47e67fc640 20 client.4229.objecter dump_active .. 0 homeless 2024-03-25T18:25:54.486 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.484+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47e0050130 con 0x7f47f00b4d60 2024-03-25T18:25:54.486 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.484+0000 7f47e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:54.486 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.484+0000 7f47e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:54.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.671+0000 7f47f7b3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:54.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.671+0000 7f47f7b3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:54.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.671+0000 7f47f7b3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:25:54.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.671+0000 7f47f7b3d640 1 -- 172.21.15.99:0/483874974 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47f0069f40 con 0x7f47f00b4d60 2024-03-25T18:25:54.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.685+0000 7f47e67fc640 1 -- 172.21.15.99:0/483874974 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f47e0052230 con 0x7f47f00b4d60 2024-03-25T18:25:54.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.685+0000 7f47e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:54.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.685+0000 7f47e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:54.687 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:25:54.687 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:54.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.687+0000 7f47f7b3d640 10 librados: watch_flush enter 2024-03-25T18:25:54.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.688+0000 7f47f7b3d640 10 librados: watch_flush exit 2024-03-25T18:25:54.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.688+0000 7f47f7b3d640 20 client.4229.objecter shutdown clearing up homeless session... 2024-03-25T18:25:54.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.688+0000 7f47f7b3d640 10 client.4229.objecter successfully canceled tick 2024-03-25T18:25:54.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.688+0000 7f47f7b3d640 1 -- 172.21.15.99:0/483874974 >> v1:172.21.15.190:6800/4220409699 conn(0x7f47d406e0b0 legacy=0x7f47d4070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.688+0000 7f47f7b3d640 10 monclient: shutdown 2024-03-25T18:25:54.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.688+0000 7f47f7b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:54.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.688+0000 7f47f7b3d640 1 -- 172.21.15.99:0/483874974 >> v1:172.21.15.99:6790/0 conn(0x7f47f00b4d60 legacy=0x7f47f0144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.688+0000 7f47f7b3d640 1 -- 172.21.15.99:0/483874974 shutdown_connections 2024-03-25T18:25:54.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.688+0000 7f47f7b3d640 1 -- 172.21.15.99:0/483874974 >> 172.21.15.99:0/483874974 conn(0x7f47f00617a0 msgr2=0x7f47f00b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:54.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.688+0000 7f47f7b3d640 1 -- 172.21.15.99:0/483874974 shutdown_connections 2024-03-25T18:25:54.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.688+0000 7f47f7b3d640 1 -- 172.21.15.99:0/483874974 wait complete. 2024-03-25T18:25:54.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:25:54.688+0000 7f47f7b3d640 1 librados: shutdown 2024-03-25T18:25:54.699 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:55.762 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-03-25T18:25:55.759+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:25:55.848 INFO:tasks.ceph.osd.6.smithi190.stderr:2024-03-25T18:25:55.845+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:25:56.130 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-03-25T18:25:56.127+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:25:57.141 INFO:tasks.ceph.osd.7.smithi190.stderr:2024-03-25T18:25:57.137+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:25:57.658 INFO:tasks.ceph.osd.2.smithi099.stderr:2024-03-25T18:25:57.653+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:25:58.221 INFO:tasks.ceph.osd.1.smithi099.stderr:2024-03-25T18:25:58.218+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:25:58.726 INFO:tasks.ceph.osd.0.smithi099.stderr:2024-03-25T18:25:58.724+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:25:58.975 INFO:tasks.ceph.osd.3.smithi099.stderr:2024-03-25T18:25:58.970+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:26:01.507 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:01.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.622+0000 7ff247c74640 10 monclient: get_monmap_and_config 2024-03-25T18:26:01.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.622+0000 7ff247c74640 10 monclient: build_initial_monmap 2024-03-25T18:26:01.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.622+0000 7ff247c74640 10 monclient: monmap: 2024-03-25T18:26:01.623 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:01.623 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:01.624 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:26:01.623310+0000 2024-03-25T18:26:01.624 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:26:01.623310+0000 2024-03-25T18:26:01.624 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:01.624 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:01.624 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:01.624 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:26:01.624 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:26:01.624 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:01.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.623+0000 7ff247c74640 1 Processor -- start 2024-03-25T18:26:01.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.623+0000 7ff247c74640 1 -- start start 2024-03-25T18:26:01.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.623+0000 7ff247c74640 10 monclient: init 2024-03-25T18:26:01.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.623+0000 7ff247c74640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:01.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.623+0000 7ff247c74640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:01.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.623+0000 7ff247c74640 10 monclient(hunting): picked mon.noname-a con 0x7ff2400a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:01.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.623+0000 7ff247c74640 10 monclient(hunting): picked mon.noname-c con 0x7ff24013f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:01.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.624+0000 7ff247c74640 10 monclient(hunting): picked mon.noname-b con 0x7ff2401430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:01.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.624+0000 7ff247c74640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff240142520 con 0x7ff2400a98f0 2024-03-25T18:26:01.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.624+0000 7ff247c74640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2400b28a0 con 0x7ff24013f710 2024-03-25T18:26:01.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.624+0000 7ff247c74640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2400b2a30 con 0x7ff2401430c0 2024-03-25T18:26:01.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.624+0000 7ff247c74640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:01.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.624+0000 7ff247c74640 10 monclient(hunting): authenticate will time out at 7379.792075s 2024-03-25T18:26:01.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.635+0000 7ff24520f640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ff24013f710 0x7ff2401429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:45170/0 (socket says 172.21.15.99:45170) 2024-03-25T18:26:01.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.635+0000 7ff24520f640 1 -- 172.21.15.99:0/3249845144 learned_addr learned my addr 172.21.15.99:0/3249845144 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:01.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.648+0000 7ff244a0e640 1 -- 172.21.15.99:0/3249845144 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200684405 0 0) 0x7ff2400b28a0 con 0x7ff24013f710 2024-03-25T18:26:01.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.648+0000 7ff244a0e640 10 monclient(hunting): _negotiate 2024-03-25T18:26:01.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.648+0000 7ff244a0e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:01.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.648+0000 7ff244a0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:01.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.648+0000 7ff244a0e640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:26:01.650 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.648+0000 7ff244a0e640 1 -- 172.21.15.99:0/3249845144 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff22c003520 con 0x7ff24013f710 2024-03-25T18:26:01.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.654+0000 7ff244a0e640 1 -- 172.21.15.99:0/3249845144 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061808629 0 0) 0x7ff2400b2a30 con 0x7ff2401430c0 2024-03-25T18:26:01.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.654+0000 7ff244a0e640 10 monclient(hunting): _negotiate 2024-03-25T18:26:01.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.654+0000 7ff244a0e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:01.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.654+0000 7ff244a0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:01.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.654+0000 7ff244a0e640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:26:01.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.654+0000 7ff244a0e640 1 -- 172.21.15.99:0/3249845144 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2400b28a0 con 0x7ff2401430c0 2024-03-25T18:26:01.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.654+0000 7ff244a0e640 1 -- 172.21.15.99:0/3249845144 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753509012 0 0) 0x7ff240142520 con 0x7ff2400a98f0 2024-03-25T18:26:01.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.654+0000 7ff244a0e640 10 monclient(hunting): _negotiate 2024-03-25T18:26:01.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.654+0000 7ff244a0e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:01.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.654+0000 7ff244a0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:01.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.654+0000 7ff244a0e640 10 monclient(hunting): my global_id is 4245 2024-03-25T18:26:01.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.654+0000 7ff244a0e640 1 -- 172.21.15.99:0/3249845144 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2400b2a30 con 0x7ff2400a98f0 2024-03-25T18:26:01.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 1 -- 172.21.15.99:0/3249845144 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 194686789 0 0) 0x7ff22c003520 con 0x7ff24013f710 2024-03-25T18:26:01.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 1 -- 172.21.15.99:0/3249845144 >> v1:172.21.15.99:6790/0 conn(0x7ff2401430c0 legacy=0x7ff24014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:01.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 1 -- 172.21.15.99:0/3249845144 >> v1:172.21.15.99:6789/0 conn(0x7ff2400a98f0 legacy=0x7ff2400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:01.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 10 monclient: _finish_hunting 0 2024-03-25T18:26:01.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 1 monclient: found mon.noname-c 2024-03-25T18:26:01.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:01.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:26:01.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 1 -- 172.21.15.99:0/3249845144 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2400b44c0 con 0x7ff24013f710 2024-03-25T18:26:01.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 10 monclient: _finish_auth 0 2024-03-25T18:26:01.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 10 monclient: _check_auth_tickets 2024-03-25T18:26:01.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:01.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 1 -- 172.21.15.99:0/3249845144 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff230001690 con 0x7ff24013f710 2024-03-25T18:26:01.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:01.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:01.658 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff244a0e640 10 monclient: dump: 2024-03-25T18:26:01.658 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:01.658 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:01.658 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:01.658 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:01.658 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:01.658 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:01.658 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:01.658 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:01.658 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:01.658 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:01.658 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff247c74640 5 monclient: authenticate success, global_id 4258 2024-03-25T18:26:01.658 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.655+0000 7ff247c74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:01.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.659+0000 7ff244a0e640 1 -- 172.21.15.99:0/3249845144 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2300017f0 con 0x7ff24013f710 2024-03-25T18:26:01.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.659+0000 7ff244a0e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:01.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.659+0000 7ff244a0e640 1 -- 172.21.15.99:0/3249845144 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff230001c00 con 0x7ff24013f710 2024-03-25T18:26:01.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.659+0000 7ff247c74640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:01.661 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.659+0000 7ff247c74640 4 set_mon_vals no callback set 2024-03-25T18:26:01.661 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.659+0000 7ff244a0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:01.661 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.659+0000 7ff244a0e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:01.661 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.659+0000 7ff244a0e640 10 monclient: dump: 2024-03-25T18:26:01.661 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:01.661 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:01.661 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:01.661 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:01.661 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:01.661 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:01.661 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:01.661 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:01.661 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:01.661 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:01.663 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.660+0000 7ff247c74640 10 monclient: shutdown 2024-03-25T18:26:01.663 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.660+0000 7ff247c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:01.663 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.660+0000 7ff247c74640 1 -- 172.21.15.99:0/3249845144 >> v1:172.21.15.190:6789/0 conn(0x7ff24013f710 legacy=0x7ff2401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:01.663 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.660+0000 7ff247c74640 1 -- 172.21.15.99:0/3249845144 shutdown_connections 2024-03-25T18:26:01.663 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.660+0000 7ff247c74640 1 -- 172.21.15.99:0/3249845144 >> 172.21.15.99:0/3249845144 conn(0x7ff2400b7ea0 msgr2=0x7ff2400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:01.663 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.660+0000 7ff247c74640 1 -- 172.21.15.99:0/3249845144 shutdown_connections 2024-03-25T18:26:01.663 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.660+0000 7ff247c74640 1 -- 172.21.15.99:0/3249845144 wait complete. 2024-03-25T18:26:01.663 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.661+0000 7ff247c74640 10 monclient: build_initial_monmap 2024-03-25T18:26:01.663 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.661+0000 7ff247c74640 10 monclient: monmap: 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.662+0000 7ff247c74640 1 librados: starting msgr at 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.662+0000 7ff247c74640 1 librados: starting objecter 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.662+0000 7ff247c74640 1 Processor -- start 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.662+0000 7ff247c74640 1 -- start start 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.662+0000 7ff247c74640 1 librados: setting wanted keys 2024-03-25T18:26:01.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.662+0000 7ff247c74640 1 librados: calling monclient init 2024-03-25T18:26:01.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.662+0000 7ff247c74640 10 monclient: init 2024-03-25T18:26:01.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.662+0000 7ff247c74640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:01.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.662+0000 7ff247c74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:01.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.662+0000 7ff247c74640 10 monclient(hunting): picked mon.noname-a con 0x7ff2400a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:01.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.662+0000 7ff247c74640 10 monclient(hunting): picked mon.noname-c con 0x7ff24013f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:01.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.662+0000 7ff247c74640 10 monclient(hunting): picked mon.noname-b con 0x7ff2401430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:01.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.662+0000 7ff247c74640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2400abdc0 con 0x7ff2400a98f0 2024-03-25T18:26:01.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.662+0000 7ff247c74640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2401d7730 con 0x7ff2401430c0 2024-03-25T18:26:01.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.663+0000 7ff247c74640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff240138f10 con 0x7ff24013f710 2024-03-25T18:26:01.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.663+0000 7ff247c74640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:01.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.663+0000 7ff247c74640 10 monclient(hunting): authenticate will time out at 7379.830916s 2024-03-25T18:26:01.683 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.683+0000 7ff246211640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ff2401430c0 0x7ff2401d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:45198/0 (socket says 172.21.15.99:45198) 2024-03-25T18:26:01.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.683+0000 7ff246211640 1 -- 172.21.15.99:0/2252324428 learned_addr learned my addr 172.21.15.99:0/2252324428 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:01.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.685+0000 7ff2367fc640 10 client.?.objecter ms_handle_connect 0x7ff2401430c0 2024-03-25T18:26:01.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.685+0000 7ff2367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:01.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.687+0000 7ff2367fc640 10 client.?.objecter ms_handle_connect 0x7ff2400a98f0 2024-03-25T18:26:01.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.687+0000 7ff2367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:01.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.690+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612348911 0 0) 0x7ff2401d7730 con 0x7ff2401430c0 2024-03-25T18:26:01.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.690+0000 7ff2367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:01.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.690+0000 7ff2367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:01.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.690+0000 7ff2367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:01.692 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.690+0000 7ff2367fc640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:26:01.692 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.690+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff224004eb0 con 0x7ff2401430c0 2024-03-25T18:26:01.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.699+0000 7ff2367fc640 10 client.?.objecter ms_handle_connect 0x7ff24013f710 2024-03-25T18:26:01.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.699+0000 7ff2367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:01.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.699+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277402475 0 0) 0x7ff2400abdc0 con 0x7ff2400a98f0 2024-03-25T18:26:01.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.699+0000 7ff2367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:01.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.699+0000 7ff2367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:01.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.699+0000 7ff2367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:01.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.699+0000 7ff2367fc640 10 monclient(hunting): my global_id is 4248 2024-03-25T18:26:01.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.699+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2401d7730 con 0x7ff2400a98f0 2024-03-25T18:26:01.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.705+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1807252841 0 0) 0x7ff224004eb0 con 0x7ff2401430c0 2024-03-25T18:26:01.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.705+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2400abdc0 con 0x7ff2401430c0 2024-03-25T18:26:01.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.705+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff238001c90 con 0x7ff2401430c0 2024-03-25T18:26:01.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.705+0000 7ff2367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:01.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.705+0000 7ff2367fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:01.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.705+0000 7ff2367fc640 10 monclient(hunting): dump: 2024-03-25T18:26:01.707 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:01.707 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:01.707 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:01.707 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:01.707 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:01.707 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:01.707 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:01.707 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:01.707 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:01.707 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:01.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.705+0000 7ff2367fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:01.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.709+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3982573424 0 0) 0x7ff2400abdc0 con 0x7ff2401430c0 2024-03-25T18:26:01.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 >> v1:172.21.15.99:6790/0 conn(0x7ff24013f710 legacy=0x7ff2401d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:01.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 >> v1:172.21.15.99:6789/0 conn(0x7ff2400a98f0 legacy=0x7ff240138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:01.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff2367fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:01.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff2367fc640 1 monclient: found mon.b 2024-03-25T18:26:01.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff2367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:01.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff2367fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:01.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2401390a0 con 0x7ff2401430c0 2024-03-25T18:26:01.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff2367fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:01.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff2367fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:01.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff2367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:01.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff247c74640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:26:01.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff247c74640 10 monclient: _renew_subs 2024-03-25T18:26:01.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff247c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:01.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff247c74640 1 -- 172.21.15.99:0/2252324428 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2401d8320 con 0x7ff2401430c0 2024-03-25T18:26:01.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff247c74640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:01.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff247c74640 10 monclient: _renew_subs 2024-03-25T18:26:01.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff247c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:01.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff247c74640 1 -- 172.21.15.99:0/2252324428 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2401d43c0 con 0x7ff2401430c0 2024-03-25T18:26:01.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.710+0000 7ff247c74640 1 librados: init done 2024-03-25T18:26:01.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.711+0000 7ff247c74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:01.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.711+0000 7ff247c74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:01.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.711+0000 7ff247c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:01.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.711+0000 7ff247c74640 1 -- 172.21.15.99:0/2252324428 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2400627b0 con 0x7ff2401430c0 2024-03-25T18:26:01.990 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.988+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff238003970 con 0x7ff2401430c0 2024-03-25T18:26:01.990 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.988+0000 7ff2367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:01.990 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.988+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff238003d00 con 0x7ff2401430c0 2024-03-25T18:26:01.990 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.988+0000 7ff2367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:01.991 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.988+0000 7ff2367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:01.991 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.988+0000 7ff2367fc640 10 monclient: dump: 2024-03-25T18:26:01.991 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:01.991 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:01.991 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:01.991 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:01.991 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:01.991 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:01.991 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:01.991 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:01.991 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:01.991 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:01.991 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.988+0000 7ff237fff640 4 set_mon_vals no callback set 2024-03-25T18:26:01.991 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.989+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7ff238003ee0 con 0x7ff2401430c0 2024-03-25T18:26:01.992 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.990+0000 7ff245a10640 10 monclient: get_auth_request con 0x7ff22406dc80 auth_method 0 2024-03-25T18:26:01.992 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.990+0000 7ff2367fc640 10 client.4261.objecter ms_handle_connect 0x7ff22406dc80 2024-03-25T18:26:01.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.994+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7ff238054b50 con 0x7ff2401430c0 2024-03-25T18:26:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.994+0000 7ff2367fc640 10 client.4261.objecter ms_dispatch 0x7ff240000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.994+0000 7ff2367fc640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.994+0000 7ff2367fc640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.994+0000 7ff2367fc640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-25T18:26:01.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.996+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff238050660 con 0x7ff2401430c0 2024-03-25T18:26:01.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.996+0000 7ff2367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:01.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:01.996+0000 7ff2367fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:02.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.227+0000 7ff247c74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:02.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.227+0000 7ff247c74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:02.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.227+0000 7ff247c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:02.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.227+0000 7ff247c74640 1 -- 172.21.15.99:0/2252324428 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff240069f40 con 0x7ff2401430c0 2024-03-25T18:26:02.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.342+0000 7ff2367fc640 1 -- 172.21.15.99:0/2252324428 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7ff238052590 con 0x7ff2401430c0 2024-03-25T18:26:02.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.342+0000 7ff2367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:02.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.342+0000 7ff2367fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:02.343 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:26:02.343 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:02.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.344+0000 7ff247c74640 10 librados: watch_flush enter 2024-03-25T18:26:02.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.344+0000 7ff247c74640 10 librados: watch_flush exit 2024-03-25T18:26:02.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.344+0000 7ff247c74640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-25T18:26:02.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.344+0000 7ff247c74640 10 client.4261.objecter successfully canceled tick 2024-03-25T18:26:02.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.344+0000 7ff247c74640 1 -- 172.21.15.99:0/2252324428 >> v1:172.21.15.190:6800/4220409699 conn(0x7ff22406dc80 legacy=0x7ff224070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.344+0000 7ff247c74640 10 monclient: shutdown 2024-03-25T18:26:02.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.344+0000 7ff247c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:02.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.344+0000 7ff247c74640 1 -- 172.21.15.99:0/2252324428 >> v1:172.21.15.190:6789/0 conn(0x7ff2401430c0 legacy=0x7ff2401d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.344+0000 7ff247c74640 1 -- 172.21.15.99:0/2252324428 shutdown_connections 2024-03-25T18:26:02.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.344+0000 7ff247c74640 1 -- 172.21.15.99:0/2252324428 >> 172.21.15.99:0/2252324428 conn(0x7ff2400b7ea0 msgr2=0x7ff240074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:02.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.344+0000 7ff247c74640 1 -- 172.21.15.99:0/2252324428 shutdown_connections 2024-03-25T18:26:02.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.344+0000 7ff247c74640 1 -- 172.21.15.99:0/2252324428 wait complete. 2024-03-25T18:26:02.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:02.344+0000 7ff247c74640 1 librados: shutdown 2024-03-25T18:26:02.355 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:09.162 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:09.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.277+0000 7fb4ecd32640 10 monclient: get_monmap_and_config 2024-03-25T18:26:09.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.277+0000 7fb4ecd32640 10 monclient: build_initial_monmap 2024-03-25T18:26:09.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.277+0000 7fb4ecd32640 10 monclient: monmap: 2024-03-25T18:26:09.278 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:09.278 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:09.279 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:26:09.278266+0000 2024-03-25T18:26:09.279 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:26:09.278266+0000 2024-03-25T18:26:09.279 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:09.279 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:09.279 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:09.279 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:26:09.279 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:26:09.279 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:09.280 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.278+0000 7fb4ecd32640 1 Processor -- start 2024-03-25T18:26:09.280 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.278+0000 7fb4ecd32640 1 -- start start 2024-03-25T18:26:09.280 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.278+0000 7fb4ecd32640 10 monclient: init 2024-03-25T18:26:09.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.278+0000 7fb4ecd32640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:09.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.278+0000 7fb4ecd32640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:09.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.278+0000 7fb4ecd32640 10 monclient(hunting): picked mon.noname-c con 0x7fb4e80a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:09.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.278+0000 7fb4ecd32640 10 monclient(hunting): picked mon.noname-a con 0x7fb4e813f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:09.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.279+0000 7fb4ecd32640 10 monclient(hunting): picked mon.noname-b con 0x7fb4e81430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:09.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.279+0000 7fb4ecd32640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e8142520 con 0x7fb4e813f710 2024-03-25T18:26:09.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.279+0000 7fb4ecd32640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e80b28a0 con 0x7fb4e80a98f0 2024-03-25T18:26:09.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.279+0000 7fb4ecd32640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e80b2a30 con 0x7fb4e81430c0 2024-03-25T18:26:09.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.279+0000 7fb4ecd32640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:09.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.279+0000 7fb4ecd32640 10 monclient(hunting): authenticate will time out at 7387.447029s 2024-03-25T18:26:09.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.286+0000 7fb4e659c640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb4e80a98f0 0x7fb4e80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:59526/0 (socket says 172.21.15.99:59526) 2024-03-25T18:26:09.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.286+0000 7fb4e659c640 1 -- 172.21.15.99:0/1434928604 learned_addr learned my addr 172.21.15.99:0/1434928604 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:09.298 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.297+0000 7fb4e5d9b640 1 -- 172.21.15.99:0/1434928604 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050205770 0 0) 0x7fb4e80b2a30 con 0x7fb4e81430c0 2024-03-25T18:26:09.298 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.297+0000 7fb4e5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:09.298 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.297+0000 7fb4e5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:09.298 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.297+0000 7fb4e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:09.298 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.297+0000 7fb4e5d9b640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:26:09.298 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.297+0000 7fb4e5d9b640 1 -- 172.21.15.99:0/1434928604 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4d4003520 con 0x7fb4e81430c0 2024-03-25T18:26:09.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.301+0000 7fb4e5d9b640 1 -- 172.21.15.99:0/1434928604 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742547369 0 0) 0x7fb4e80b28a0 con 0x7fb4e80a98f0 2024-03-25T18:26:09.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.301+0000 7fb4e5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:09.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.301+0000 7fb4e5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:09.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.301+0000 7fb4e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:09.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.301+0000 7fb4e5d9b640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:26:09.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.301+0000 7fb4e5d9b640 1 -- 172.21.15.99:0/1434928604 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e80b2a30 con 0x7fb4e80a98f0 2024-03-25T18:26:09.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.302+0000 7fb4e5d9b640 1 -- 172.21.15.99:0/1434928604 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549860435 0 0) 0x7fb4e8142520 con 0x7fb4e813f710 2024-03-25T18:26:09.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.302+0000 7fb4e5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:09.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.302+0000 7fb4e5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:09.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.302+0000 7fb4e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:09.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.302+0000 7fb4e5d9b640 10 monclient(hunting): my global_id is 4251 2024-03-25T18:26:09.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.302+0000 7fb4e5d9b640 1 -- 172.21.15.99:0/1434928604 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e80b28a0 con 0x7fb4e813f710 2024-03-25T18:26:09.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 1 -- 172.21.15.99:0/1434928604 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2142004775 0 0) 0x7fb4e80b28a0 con 0x7fb4e813f710 2024-03-25T18:26:09.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 1 -- 172.21.15.99:0/1434928604 >> v1:172.21.15.99:6790/0 conn(0x7fb4e81430c0 legacy=0x7fb4e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:09.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 1 -- 172.21.15.99:0/1434928604 >> v1:172.21.15.190:6789/0 conn(0x7fb4e80a98f0 legacy=0x7fb4e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:09.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:26:09.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 1 monclient: found mon.noname-a 2024-03-25T18:26:09.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:09.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:26:09.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 1 -- 172.21.15.99:0/1434928604 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e80b44c0 con 0x7fb4e813f710 2024-03-25T18:26:09.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:26:09.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:26:09.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:09.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 1 -- 172.21.15.99:0/1434928604 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb4d00015b0 con 0x7fb4e813f710 2024-03-25T18:26:09.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:09.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:09.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4e5d9b640 10 monclient: dump: 2024-03-25T18:26:09.319 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:09.319 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:09.319 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:09.319 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:09.319 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:09.319 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:09.319 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:09.319 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:09.319 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:09.319 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:09.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4ecd32640 5 monclient: authenticate success, global_id 4251 2024-03-25T18:26:09.320 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.316+0000 7fb4ecd32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:09.328 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.327+0000 7fb4e5d9b640 1 -- 172.21.15.99:0/1434928604 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4d0001710 con 0x7fb4e813f710 2024-03-25T18:26:09.328 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.327+0000 7fb4e5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:09.328 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.327+0000 7fb4e5d9b640 1 -- 172.21.15.99:0/1434928604 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb4d0001b60 con 0x7fb4e813f710 2024-03-25T18:26:09.328 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.327+0000 7fb4e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:09.328 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.327+0000 7fb4e5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:09.328 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.327+0000 7fb4e5d9b640 10 monclient: dump: 2024-03-25T18:26:09.328 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:09.328 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:09.328 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:09.328 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:09.328 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:09.329 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:09.329 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:09.329 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:09.329 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:09.329 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:09.329 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.327+0000 7fb4ecd32640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:09.329 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.327+0000 7fb4ecd32640 4 set_mon_vals no callback set 2024-03-25T18:26:09.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.328+0000 7fb4ecd32640 10 monclient: shutdown 2024-03-25T18:26:09.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.328+0000 7fb4ecd32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:09.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.328+0000 7fb4ecd32640 1 -- 172.21.15.99:0/1434928604 >> v1:172.21.15.99:6789/0 conn(0x7fb4e813f710 legacy=0x7fb4e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:09.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.328+0000 7fb4ecd32640 1 -- 172.21.15.99:0/1434928604 shutdown_connections 2024-03-25T18:26:09.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.328+0000 7fb4ecd32640 1 -- 172.21.15.99:0/1434928604 >> 172.21.15.99:0/1434928604 conn(0x7fb4e80b7ea0 msgr2=0x7fb4e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:09.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.328+0000 7fb4ecd32640 1 -- 172.21.15.99:0/1434928604 shutdown_connections 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.328+0000 7fb4ecd32640 1 -- 172.21.15.99:0/1434928604 wait complete. 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.329+0000 7fb4ecd32640 10 monclient: build_initial_monmap 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.329+0000 7fb4ecd32640 10 monclient: monmap: 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.329+0000 7fb4ecd32640 1 librados: starting msgr at 2024-03-25T18:26:09.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.329+0000 7fb4ecd32640 1 librados: starting objecter 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.329+0000 7fb4ecd32640 1 Processor -- start 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.329+0000 7fb4ecd32640 1 -- start start 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.329+0000 7fb4ecd32640 1 librados: setting wanted keys 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.329+0000 7fb4ecd32640 1 librados: calling monclient init 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.329+0000 7fb4ecd32640 10 monclient: init 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.330+0000 7fb4ecd32640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.330+0000 7fb4ecd32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.330+0000 7fb4ecd32640 10 monclient(hunting): picked mon.noname-b con 0x7fb4e80a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.330+0000 7fb4ecd32640 10 monclient(hunting): picked mon.noname-c con 0x7fb4e813f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.330+0000 7fb4ecd32640 10 monclient(hunting): picked mon.noname-a con 0x7fb4e81430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.330+0000 7fb4ecd32640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e81d7830 con 0x7fb4e81430c0 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.330+0000 7fb4ecd32640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e80ab9e0 con 0x7fb4e80a98f0 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.330+0000 7fb4ecd32640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e81d7500 con 0x7fb4e813f710 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.330+0000 7fb4ecd32640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:09.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.330+0000 7fb4ecd32640 10 monclient(hunting): authenticate will time out at 7387.498283s 2024-03-25T18:26:09.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.336+0000 7fb4e6d9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fb4e81430c0 0x7fb4e81e1f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:48476/0 (socket says 172.21.15.99:48476) 2024-03-25T18:26:09.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.336+0000 7fb4e6d9d640 1 -- 172.21.15.99:0/2233763379 learned_addr learned my addr 172.21.15.99:0/2233763379 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:09.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.338+0000 7fb4deffd640 10 client.?.objecter ms_handle_connect 0x7fb4e81430c0 2024-03-25T18:26:09.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.338+0000 7fb4deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:09.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.342+0000 7fb4deffd640 10 client.?.objecter ms_handle_connect 0x7fb4e813f710 2024-03-25T18:26:09.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.342+0000 7fb4deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:09.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.344+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53521105 0 0) 0x7fb4e81d7830 con 0x7fb4e81430c0 2024-03-25T18:26:09.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.344+0000 7fb4deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:09.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.344+0000 7fb4deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:09.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.344+0000 7fb4deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:09.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.344+0000 7fb4deffd640 10 monclient(hunting): my global_id is 4254 2024-03-25T18:26:09.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.344+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4c8004eb0 con 0x7fb4e81430c0 2024-03-25T18:26:09.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.346+0000 7fb4deffd640 10 client.?.objecter ms_handle_connect 0x7fb4e80a98f0 2024-03-25T18:26:09.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.346+0000 7fb4deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:09.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.352+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096107715 0 0) 0x7fb4e81d7500 con 0x7fb4e813f710 2024-03-25T18:26:09.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.352+0000 7fb4deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:09.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.352+0000 7fb4deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:09.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.352+0000 7fb4deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:09.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.352+0000 7fb4deffd640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:26:09.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.352+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e81d7830 con 0x7fb4e813f710 2024-03-25T18:26:09.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.358+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172237351 0 0) 0x7fb4e80ab9e0 con 0x7fb4e80a98f0 2024-03-25T18:26:09.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.358+0000 7fb4deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:09.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.358+0000 7fb4deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:09.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.358+0000 7fb4deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:09.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.358+0000 7fb4deffd640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:26:09.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.358+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e81d7500 con 0x7fb4e80a98f0 2024-03-25T18:26:09.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.367+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4262600546 0 0) 0x7fb4c8004eb0 con 0x7fb4e81430c0 2024-03-25T18:26:09.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.367+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4e80ab9e0 con 0x7fb4e81430c0 2024-03-25T18:26:09.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.367+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb4e0001c30 con 0x7fb4e81430c0 2024-03-25T18:26:09.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.367+0000 7fb4deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:09.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.367+0000 7fb4deffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:09.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.367+0000 7fb4deffd640 10 monclient(hunting): dump: 2024-03-25T18:26:09.369 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:09.369 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:09.369 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:09.369 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:09.369 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:09.369 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:09.369 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:09.369 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:09.369 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:09.369 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:09.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.367+0000 7fb4deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:09.381 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.379+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3880971364 0 0) 0x7fb4e81d7500 con 0x7fb4e80a98f0 2024-03-25T18:26:09.381 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.380+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4c8004eb0 con 0x7fb4e80a98f0 2024-03-25T18:26:09.381 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.380+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb4d8001930 con 0x7fb4e80a98f0 2024-03-25T18:26:09.381 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.380+0000 7fb4deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:09.381 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.380+0000 7fb4deffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:09.381 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.380+0000 7fb4deffd640 10 monclient(hunting): dump: 2024-03-25T18:26:09.381 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:09.381 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:09.381 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:09.382 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:09.382 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:09.382 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:09.382 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:09.382 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:09.382 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:09.382 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:09.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.394+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3977557894 0 0) 0x7fb4e81d7830 con 0x7fb4e813f710 2024-03-25T18:26:09.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.394+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4e81d7500 con 0x7fb4e813f710 2024-03-25T18:26:09.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.394+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb4d0001a20 con 0x7fb4e813f710 2024-03-25T18:26:09.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.394+0000 7fb4deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:09.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.394+0000 7fb4deffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:09.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.394+0000 7fb4deffd640 10 monclient(hunting): dump: 2024-03-25T18:26:09.395 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:09.395 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:09.396 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:09.396 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:09.396 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:09.396 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:09.396 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:09.396 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:09.396 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:09.396 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:09.398 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1865206691 0 0) 0x7fb4e80ab9e0 con 0x7fb4e81430c0 2024-03-25T18:26:09.398 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 >> v1:172.21.15.99:6790/0 conn(0x7fb4e813f710 legacy=0x7fb4e81d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:09.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 >> v1:172.21.15.190:6789/0 conn(0x7fb4e80a98f0 legacy=0x7fb4e8138760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:09.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4deffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:09.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4deffd640 1 monclient: found mon.a 2024-03-25T18:26:09.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:09.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4deffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:26:09.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e81d83d0 con 0x7fb4e81430c0 2024-03-25T18:26:09.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4deffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:09.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4deffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:09.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:09.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4ecd32640 5 monclient: authenticate success, global_id 4254 2024-03-25T18:26:09.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4ecd32640 10 monclient: _renew_subs 2024-03-25T18:26:09.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4ecd32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:26:09.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4ecd32640 1 -- 172.21.15.99:0/2233763379 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e81379c0 con 0x7fb4e81430c0 2024-03-25T18:26:09.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4ecd32640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:09.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4ecd32640 10 monclient: _renew_subs 2024-03-25T18:26:09.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4ecd32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:26:09.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4ecd32640 1 -- 172.21.15.99:0/2233763379 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e81351f0 con 0x7fb4e81430c0 2024-03-25T18:26:09.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.397+0000 7fb4ecd32640 1 librados: init done 2024-03-25T18:26:09.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.398+0000 7fb4ecd32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:09.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.398+0000 7fb4ecd32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:09.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.398+0000 7fb4ecd32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:26:09.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.398+0000 7fb4ecd32640 1 -- 172.21.15.99:0/2233763379 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e8137cc0 con 0x7fb4e81430c0 2024-03-25T18:26:09.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.686+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4e00020e0 con 0x7fb4e81430c0 2024-03-25T18:26:09.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.686+0000 7fb4deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:09.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.686+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb4e0003b30 con 0x7fb4e81430c0 2024-03-25T18:26:09.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.686+0000 7fb4deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:09.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.686+0000 7fb4deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:09.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.686+0000 7fb4deffd640 10 monclient: dump: 2024-03-25T18:26:09.688 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:09.688 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:09.688 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:09.688 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:09.689 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:09.689 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:09.689 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:09.689 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:09.689 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:09.689 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:09.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.686+0000 7fb4e559a640 4 set_mon_vals no callback set 2024-03-25T18:26:09.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.687+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fb4e0003d50 con 0x7fb4e81430c0 2024-03-25T18:26:09.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.688+0000 7fb4e659c640 10 monclient: get_auth_request con 0x7fb4c806e0f0 auth_method 0 2024-03-25T18:26:09.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.688+0000 7fb4deffd640 10 client.4254.objecter ms_handle_connect 0x7fb4c806e0f0 2024-03-25T18:26:09.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.693+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fb4e0054a50 con 0x7fb4e81430c0 2024-03-25T18:26:09.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.693+0000 7fb4deffd640 10 client.4254.objecter ms_dispatch 0x7fb4e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:09.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.693+0000 7fb4deffd640 3 client.4254.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:09.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.693+0000 7fb4deffd640 3 client.4254.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:09.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.693+0000 7fb4deffd640 20 client.4254.objecter dump_active .. 0 homeless 2024-03-25T18:26:09.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.693+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4e0050560 con 0x7fb4e81430c0 2024-03-25T18:26:09.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.693+0000 7fb4deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:09.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.693+0000 7fb4deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.939+0000 7fb4ecd32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.939+0000 7fb4ecd32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.939+0000 7fb4ecd32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:26:09.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:09.939+0000 7fb4ecd32640 1 -- 172.21.15.99:0/2233763379 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4e81d79e0 con 0x7fb4e81430c0 2024-03-25T18:26:10.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.077+0000 7fb4deffd640 1 -- 172.21.15.99:0/2233763379 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fb4e81d79e0 con 0x7fb4e81430c0 2024-03-25T18:26:10.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.077+0000 7fb4deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:10.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.077+0000 7fb4deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:10.078 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:26:10.079 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:10.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.079+0000 7fb4ecd32640 10 librados: watch_flush enter 2024-03-25T18:26:10.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.079+0000 7fb4ecd32640 10 librados: watch_flush exit 2024-03-25T18:26:10.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.079+0000 7fb4ecd32640 20 client.4254.objecter shutdown clearing up homeless session... 2024-03-25T18:26:10.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.079+0000 7fb4ecd32640 10 client.4254.objecter successfully canceled tick 2024-03-25T18:26:10.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.079+0000 7fb4ecd32640 1 -- 172.21.15.99:0/2233763379 >> v1:172.21.15.190:6800/4220409699 conn(0x7fb4c806e0f0 legacy=0x7fb4c80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.079+0000 7fb4ecd32640 10 monclient: shutdown 2024-03-25T18:26:10.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.079+0000 7fb4ecd32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:10.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.079+0000 7fb4ecd32640 1 -- 172.21.15.99:0/2233763379 >> v1:172.21.15.99:6789/0 conn(0x7fb4e81430c0 legacy=0x7fb4e81e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.079+0000 7fb4ecd32640 1 -- 172.21.15.99:0/2233763379 shutdown_connections 2024-03-25T18:26:10.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.079+0000 7fb4ecd32640 1 -- 172.21.15.99:0/2233763379 >> 172.21.15.99:0/2233763379 conn(0x7fb4e80b7ea0 msgr2=0x7fb4e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:10.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.079+0000 7fb4ecd32640 1 -- 172.21.15.99:0/2233763379 shutdown_connections 2024-03-25T18:26:10.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.079+0000 7fb4ecd32640 1 -- 172.21.15.99:0/2233763379 wait complete. 2024-03-25T18:26:10.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:10.079+0000 7fb4ecd32640 1 librados: shutdown 2024-03-25T18:26:10.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:13.558 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:01:14.151 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:13.658 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:01:14.250 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:13.805 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:14.364 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:13.824 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:01:14.416 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:13.864 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:14.423 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:13.893 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:01:14.486 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:13.917 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:01:14.509 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:14.077 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:14.635 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:14.130 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:01:14.723 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:16.899 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:16.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.994+0000 7fb65d201640 10 monclient: get_monmap_and_config 2024-03-25T18:26:16.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.994+0000 7fb65d201640 10 monclient: build_initial_monmap 2024-03-25T18:26:16.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.994+0000 7fb65d201640 10 monclient: monmap: 2024-03-25T18:26:16.996 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:16.996 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:16.996 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:26:16.995432+0000 2024-03-25T18:26:16.996 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:26:16.995432+0000 2024-03-25T18:26:16.996 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:16.996 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:16.996 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:16.996 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:26:16.996 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:26:16.996 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:16.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.995+0000 7fb65d201640 1 Processor -- start 2024-03-25T18:26:16.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.995+0000 7fb65d201640 1 -- start start 2024-03-25T18:26:16.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.995+0000 7fb65d201640 10 monclient: init 2024-03-25T18:26:16.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.995+0000 7fb65d201640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:16.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.995+0000 7fb65d201640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:16.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.995+0000 7fb65d201640 10 monclient(hunting): picked mon.noname-b con 0x7fb6580ae6e0 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:16.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.995+0000 7fb65d201640 10 monclient(hunting): picked mon.noname-a con 0x7fb65806f080 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:16.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.995+0000 7fb65d201640 10 monclient(hunting): picked mon.noname-c con 0x7fb6580704e0 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:16.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.995+0000 7fb65d201640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6580718d0 con 0x7fb65806f080 2024-03-25T18:26:16.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.995+0000 7fb65d201640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65806fa30 con 0x7fb6580704e0 2024-03-25T18:26:16.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.995+0000 7fb65d201640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb658070090 con 0x7fb6580ae6e0 2024-03-25T18:26:16.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.995+0000 7fb65d201640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:16.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:16.995+0000 7fb65d201640 10 monclient(hunting): authenticate will time out at 7395.163699s 2024-03-25T18:26:17.003 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.003+0000 7fb65659c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fb65806f080 0x7fb658071c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45504/0 (socket says 172.21.15.99:45504) 2024-03-25T18:26:17.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.003+0000 7fb65659c640 1 -- 172.21.15.99:0/3889429022 learned_addr learned my addr 172.21.15.99:0/3889429022 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:17.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.020+0000 7fb655d9b640 1 -- 172.21.15.99:0/3889429022 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033080850 0 0) 0x7fb6580718d0 con 0x7fb65806f080 2024-03-25T18:26:17.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.020+0000 7fb655d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.020+0000 7fb655d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.020+0000 7fb655d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.020+0000 7fb655d9b640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:26:17.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.020+0000 7fb655d9b640 1 -- 172.21.15.99:0/3889429022 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb640003520 con 0x7fb65806f080 2024-03-25T18:26:17.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.025+0000 7fb655d9b640 1 -- 172.21.15.99:0/3889429022 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4122531009 0 0) 0x7fb640003520 con 0x7fb65806f080 2024-03-25T18:26:17.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.025+0000 7fb655d9b640 1 -- 172.21.15.99:0/3889429022 >> v1:172.21.15.99:6790/0 conn(0x7fb6580ae6e0 legacy=0x7fb65813c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.025+0000 7fb655d9b640 1 -- 172.21.15.99:0/3889429022 >> v1:172.21.15.190:6789/0 conn(0x7fb6580704e0 legacy=0x7fb65814a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:26:17.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.025+0000 7fb655d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:26:17.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.025+0000 7fb655d9b640 1 monclient: found mon.noname-a 2024-03-25T18:26:17.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.025+0000 7fb655d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:17.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.025+0000 7fb655d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:26:17.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.025+0000 7fb655d9b640 1 -- 172.21.15.99:0/3889429022 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65806f510 con 0x7fb65806f080 2024-03-25T18:26:17.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.025+0000 7fb655d9b640 10 monclient: _finish_auth 0 2024-03-25T18:26:17.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.025+0000 7fb655d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:26:17.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.025+0000 7fb655d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:17.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.025+0000 7fb65d201640 5 monclient: authenticate success, global_id 4257 2024-03-25T18:26:17.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.025+0000 7fb65d201640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:17.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.029+0000 7fb655d9b640 1 -- 172.21.15.99:0/3889429022 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb64c001690 con 0x7fb65806f080 2024-03-25T18:26:17.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.029+0000 7fb655d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.029+0000 7fb655d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:17.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.029+0000 7fb655d9b640 10 monclient: dump: 2024-03-25T18:26:17.030 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:17.030 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:17.030 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:17.030 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:17.030 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.031 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:17.031 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:17.031 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:17.031 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:17.031 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:17.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.029+0000 7fb65d201640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:17.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.046+0000 7fb655d9b640 1 -- 172.21.15.99:0/3889429022 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb64c001960 con 0x7fb65806f080 2024-03-25T18:26:17.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.046+0000 7fb655d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:17.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.046+0000 7fb655d9b640 1 -- 172.21.15.99:0/3889429022 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb64c001cf0 con 0x7fb65806f080 2024-03-25T18:26:17.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.046+0000 7fb65d201640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:17.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.046+0000 7fb65d201640 4 set_mon_vals no callback set 2024-03-25T18:26:17.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.046+0000 7fb655d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.046+0000 7fb655d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:17.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.046+0000 7fb655d9b640 10 monclient: dump: 2024-03-25T18:26:17.047 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:17.047 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:17.047 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:17.048 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:17.048 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.048 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:17.048 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:17.048 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:17.048 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:17.048 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:17.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.047+0000 7fb65d201640 10 monclient: shutdown 2024-03-25T18:26:17.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.047+0000 7fb65d201640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:17.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.047+0000 7fb65d201640 1 -- 172.21.15.99:0/3889429022 >> v1:172.21.15.99:6789/0 conn(0x7fb65806f080 legacy=0x7fb658071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.047+0000 7fb65d201640 1 -- 172.21.15.99:0/3889429022 shutdown_connections 2024-03-25T18:26:17.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.047+0000 7fb65d201640 1 -- 172.21.15.99:0/3889429022 >> 172.21.15.99:0/3889429022 conn(0x7fb65806b630 msgr2=0x7fb6580b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:17.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.047+0000 7fb65d201640 1 -- 172.21.15.99:0/3889429022 shutdown_connections 2024-03-25T18:26:17.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.047+0000 7fb65d201640 1 -- 172.21.15.99:0/3889429022 wait complete. 2024-03-25T18:26:17.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.048+0000 7fb65d201640 10 monclient: build_initial_monmap 2024-03-25T18:26:17.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.048+0000 7fb65d201640 10 monclient: monmap: 2024-03-25T18:26:17.050 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:17.050 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:17.050 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:26:17.050 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:26:17.050 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:17.050 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:26:17.050 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:17.050 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:26:17.050 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:26:17.050 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:17.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.048+0000 7fb65d201640 1 librados: starting msgr at 2024-03-25T18:26:17.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.048+0000 7fb65d201640 1 librados: starting objecter 2024-03-25T18:26:17.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.049+0000 7fb65d201640 1 Processor -- start 2024-03-25T18:26:17.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.049+0000 7fb65d201640 1 -- start start 2024-03-25T18:26:17.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.049+0000 7fb65d201640 1 librados: setting wanted keys 2024-03-25T18:26:17.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.049+0000 7fb65d201640 1 librados: calling monclient init 2024-03-25T18:26:17.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.049+0000 7fb65d201640 10 monclient: init 2024-03-25T18:26:17.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.049+0000 7fb65d201640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:17.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.049+0000 7fb65d201640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:17.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.049+0000 7fb65d201640 10 monclient(hunting): picked mon.noname-c con 0x7fb6580704e0 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:17.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.050+0000 7fb65d201640 10 monclient(hunting): picked mon.noname-a con 0x7fb6580ae6e0 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:17.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.050+0000 7fb65d201640 10 monclient(hunting): picked mon.noname-b con 0x7fb6581db680 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:17.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.050+0000 7fb65d201640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65813cdf0 con 0x7fb6580ae6e0 2024-03-25T18:26:17.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.050+0000 7fb65d201640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb65813dcf0 con 0x7fb6581db680 2024-03-25T18:26:17.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.050+0000 7fb65d201640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb658141560 con 0x7fb6580704e0 2024-03-25T18:26:17.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.050+0000 7fb65d201640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:17.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.050+0000 7fb65d201640 10 monclient(hunting): authenticate will time out at 7395.218245s 2024-03-25T18:26:17.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.059+0000 7fb65759e640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb6581db680 0x7fb6581e2d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:34056/0 (socket says 172.21.15.99:34056) 2024-03-25T18:26:17.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.059+0000 7fb65759e640 1 -- 172.21.15.99:0/120703639 learned_addr learned my addr 172.21.15.99:0/120703639 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:17.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.059+0000 7fb63f7fe640 10 client.?.objecter ms_handle_connect 0x7fb6581db680 2024-03-25T18:26:17.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.059+0000 7fb63f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:17.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.063+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513652430 0 0) 0x7fb65813dcf0 con 0x7fb6581db680 2024-03-25T18:26:17.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.063+0000 7fb63f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.063+0000 7fb63f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.063+0000 7fb63f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.063+0000 7fb63f7fe640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:26:17.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.063+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb644004eb0 con 0x7fb6581db680 2024-03-25T18:26:17.066 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.065+0000 7fb63f7fe640 10 client.?.objecter ms_handle_connect 0x7fb6580704e0 2024-03-25T18:26:17.066 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.065+0000 7fb63f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:17.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.069+0000 7fb63f7fe640 10 client.?.objecter ms_handle_connect 0x7fb6580ae6e0 2024-03-25T18:26:17.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.069+0000 7fb63f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:17.073 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.073+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308433692 0 0) 0x7fb658141560 con 0x7fb6580704e0 2024-03-25T18:26:17.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.073+0000 7fb63f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.073+0000 7fb63f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.073+0000 7fb63f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.073+0000 7fb63f7fe640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:26:17.074 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.073+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb65813dcf0 con 0x7fb6580704e0 2024-03-25T18:26:17.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.075+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364915824 0 0) 0x7fb65813cdf0 con 0x7fb6580ae6e0 2024-03-25T18:26:17.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.075+0000 7fb63f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.075+0000 7fb63f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.075+0000 7fb63f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.075+0000 7fb63f7fe640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:26:17.077 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.075+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb658141560 con 0x7fb6580ae6e0 2024-03-25T18:26:17.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.076+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 999220211 0 0) 0x7fb644004eb0 con 0x7fb6581db680 2024-03-25T18:26:17.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.076+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb65813cdf0 con 0x7fb6581db680 2024-03-25T18:26:17.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.077+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb650001c60 con 0x7fb6581db680 2024-03-25T18:26:17.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.077+0000 7fb63f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.077+0000 7fb63f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:17.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.077+0000 7fb63f7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:17.078 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:17.078 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:17.078 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:17.078 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:17.079 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.079 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:17.079 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:17.079 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:17.079 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:17.079 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:17.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.077+0000 7fb63f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:17.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.086+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 87533181 0 0) 0x7fb65813dcf0 con 0x7fb6580704e0 2024-03-25T18:26:17.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.086+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb644004eb0 con 0x7fb6580704e0 2024-03-25T18:26:17.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.086+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb648001800 con 0x7fb6580704e0 2024-03-25T18:26:17.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.086+0000 7fb63f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.086+0000 7fb63f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:17.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.086+0000 7fb63f7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:17.087 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:17.087 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:17.087 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:17.088 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:17.088 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.088 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:17.088 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:17.088 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:17.088 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:17.088 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:17.095 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1395839122 0 0) 0x7fb65813cdf0 con 0x7fb6581db680 2024-03-25T18:26:17.095 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 >> v1:172.21.15.99:6790/0 conn(0x7fb6580704e0 legacy=0x7fb65813e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.095 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 >> v1:172.21.15.99:6789/0 conn(0x7fb6580ae6e0 legacy=0x7fb658135be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.095 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb63f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:17.095 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb63f7fe640 1 monclient: found mon.b 2024-03-25T18:26:17.095 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb63f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:17.095 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb63f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:17.095 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6581416f0 con 0x7fb6581db680 2024-03-25T18:26:17.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb63f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:17.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb63f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:17.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb63f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:17.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb65d201640 5 monclient: authenticate success, global_id 4270 2024-03-25T18:26:17.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb65d201640 10 monclient: _renew_subs 2024-03-25T18:26:17.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb65d201640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:17.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb65d201640 1 -- 172.21.15.99:0/120703639 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb65813e920 con 0x7fb6581db680 2024-03-25T18:26:17.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb65d201640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:17.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb65d201640 10 monclient: _renew_subs 2024-03-25T18:26:17.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb65d201640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:17.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb65d201640 1 -- 172.21.15.99:0/120703639 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65813d920 con 0x7fb6581db680 2024-03-25T18:26:17.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.094+0000 7fb65d201640 1 librados: init done 2024-03-25T18:26:17.097 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.095+0000 7fb65d201640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:17.097 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.095+0000 7fb65d201640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:17.097 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.095+0000 7fb65d201640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:17.097 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.095+0000 7fb65d201640 1 -- 172.21.15.99:0/120703639 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6580627b0 con 0x7fb6581db680 2024-03-25T18:26:17.382 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.381+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb650003970 con 0x7fb6581db680 2024-03-25T18:26:17.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.381+0000 7fb63f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:17.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.381+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb650003cc0 con 0x7fb6581db680 2024-03-25T18:26:17.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.381+0000 7fb63f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.381+0000 7fb63f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:17.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.381+0000 7fb63f7fe640 10 monclient: dump: 2024-03-25T18:26:17.383 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:17.383 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:17.383 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:17.383 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:17.383 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.383 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:17.383 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:17.383 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:17.383 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:17.384 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:17.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.381+0000 7fb65559a640 4 set_mon_vals no callback set 2024-03-25T18:26:17.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.382+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fb650003ee0 con 0x7fb6581db680 2024-03-25T18:26:17.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.383+0000 7fb656d9d640 10 monclient: get_auth_request con 0x7fb64406df60 auth_method 0 2024-03-25T18:26:17.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.383+0000 7fb63f7fe640 10 client.4270.objecter ms_handle_connect 0x7fb64406df60 2024-03-25T18:26:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.387+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fb650054a00 con 0x7fb6581db680 2024-03-25T18:26:17.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.387+0000 7fb63f7fe640 10 client.4270.objecter ms_dispatch 0x7fb658000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:17.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.387+0000 7fb63f7fe640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:17.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.387+0000 7fb63f7fe640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:17.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.387+0000 7fb63f7fe640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T18:26:17.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.388+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb650050510 con 0x7fb6581db680 2024-03-25T18:26:17.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.388+0000 7fb63f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:17.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.388+0000 7fb63f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:17.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.635+0000 7fb65d201640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:17.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.635+0000 7fb65d201640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:17.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.635+0000 7fb65d201640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:17.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.635+0000 7fb65d201640 1 -- 172.21.15.99:0/120703639 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb65813de80 con 0x7fb6581db680 2024-03-25T18:26:17.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.673+0000 7fb63f7fe640 1 -- 172.21.15.99:0/120703639 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fb650052440 con 0x7fb6581db680 2024-03-25T18:26:17.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.673+0000 7fb63f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:17.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.673+0000 7fb63f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:17.675 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:26:17.675 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:17.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.675+0000 7fb65d201640 10 librados: watch_flush enter 2024-03-25T18:26:17.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.675+0000 7fb65d201640 10 librados: watch_flush exit 2024-03-25T18:26:17.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.675+0000 7fb65d201640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T18:26:17.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.675+0000 7fb65d201640 10 client.4270.objecter successfully canceled tick 2024-03-25T18:26:17.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.675+0000 7fb65d201640 1 -- 172.21.15.99:0/120703639 >> v1:172.21.15.190:6800/4220409699 conn(0x7fb64406df60 legacy=0x7fb644070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.675+0000 7fb65d201640 10 monclient: shutdown 2024-03-25T18:26:17.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.675+0000 7fb65d201640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:17.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.675+0000 7fb65d201640 1 -- 172.21.15.99:0/120703639 >> v1:172.21.15.190:6789/0 conn(0x7fb6581db680 legacy=0x7fb6581e2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.676+0000 7fb65d201640 1 -- 172.21.15.99:0/120703639 shutdown_connections 2024-03-25T18:26:17.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.676+0000 7fb65d201640 1 -- 172.21.15.99:0/120703639 >> 172.21.15.99:0/120703639 conn(0x7fb65806b630 msgr2=0x7fb6580af590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:17.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.676+0000 7fb65d201640 1 -- 172.21.15.99:0/120703639 shutdown_connections 2024-03-25T18:26:17.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.676+0000 7fb65d201640 1 -- 172.21.15.99:0/120703639 wait complete. 2024-03-25T18:26:17.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:17.676+0000 7fb65d201640 1 librados: shutdown 2024-03-25T18:26:17.689 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:24.496 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:24.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.609+0000 7f8629d32640 10 monclient: get_monmap_and_config 2024-03-25T18:26:24.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.609+0000 7f8629d32640 10 monclient: build_initial_monmap 2024-03-25T18:26:24.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.609+0000 7f8629d32640 10 monclient: monmap: 2024-03-25T18:26:24.610 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:24.610 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:24.610 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:26:24.610241+0000 2024-03-25T18:26:24.610 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:26:24.610241+0000 2024-03-25T18:26:24.610 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:24.610 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:24.611 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:24.611 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:26:24.611 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:26:24.611 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:24.612 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.610+0000 7f8629d32640 1 Processor -- start 2024-03-25T18:26:24.612 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.610+0000 7f8629d32640 1 -- start start 2024-03-25T18:26:24.612 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.610+0000 7f8629d32640 10 monclient: init 2024-03-25T18:26:24.612 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.610+0000 7f8629d32640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:24.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.610+0000 7f8629d32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:24.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.610+0000 7f8629d32640 10 monclient(hunting): picked mon.noname-c con 0x7f86240a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:24.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.611+0000 7f8629d32640 10 monclient(hunting): picked mon.noname-b con 0x7f862413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:24.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.611+0000 7f8629d32640 10 monclient(hunting): picked mon.noname-a con 0x7f86241430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:24.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.611+0000 7f8629d32640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8624142520 con 0x7f86241430c0 2024-03-25T18:26:24.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.611+0000 7f8629d32640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86240b28a0 con 0x7f86240a98f0 2024-03-25T18:26:24.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.611+0000 7f8629d32640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86240b2a30 con 0x7f862413f710 2024-03-25T18:26:24.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.611+0000 7f8629d32640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:24.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.611+0000 7f8629d32640 10 monclient(hunting): authenticate will time out at 7402.778989s 2024-03-25T18:26:24.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.620+0000 7f86237fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f86240a98f0 0x7f86240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:57146/0 (socket says 172.21.15.99:57146) 2024-03-25T18:26:24.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.620+0000 7f86237fe640 1 -- 172.21.15.99:0/551684267 learned_addr learned my addr 172.21.15.99:0/551684267 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:24.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.645+0000 7f86227fc640 1 -- 172.21.15.99:0/551684267 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385354804 0 0) 0x7f86240b28a0 con 0x7f86240a98f0 2024-03-25T18:26:24.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.645+0000 7f86227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:24.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.645+0000 7f86227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:24.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.645+0000 7f86227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:24.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.645+0000 7f86227fc640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:26:24.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.645+0000 7f86227fc640 1 -- 172.21.15.99:0/551684267 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f860c003520 con 0x7f86240a98f0 2024-03-25T18:26:24.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.654+0000 7f86227fc640 1 -- 172.21.15.99:0/551684267 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181149487 0 0) 0x7f86240b2a30 con 0x7f862413f710 2024-03-25T18:26:24.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.654+0000 7f86227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:24.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.654+0000 7f86227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:24.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.654+0000 7f86227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:24.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.654+0000 7f86227fc640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:26:24.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.654+0000 7f86227fc640 1 -- 172.21.15.99:0/551684267 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86240b28a0 con 0x7f862413f710 2024-03-25T18:26:24.658 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 1 -- 172.21.15.99:0/551684267 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 856154326 0 0) 0x7f860c003520 con 0x7f86240a98f0 2024-03-25T18:26:24.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 1 -- 172.21.15.99:0/551684267 >> v1:172.21.15.99:6790/0 conn(0x7f862413f710 legacy=0x7f86241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:24.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 1 -- 172.21.15.99:0/551684267 >> v1:172.21.15.99:6789/0 conn(0x7f86241430c0 legacy=0x7f862414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:24.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:24.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 1 monclient: found mon.noname-c 2024-03-25T18:26:24.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:24.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:26:24.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 1 -- 172.21.15.99:0/551684267 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86240b44c0 con 0x7f86240a98f0 2024-03-25T18:26:24.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:24.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:24.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:24.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 1 -- 172.21.15.99:0/551684267 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8614001690 con 0x7f86240a98f0 2024-03-25T18:26:24.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f8629d32640 5 monclient: authenticate success, global_id 4273 2024-03-25T18:26:24.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f8629d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:24.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:24.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:24.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f86227fc640 10 monclient: dump: 2024-03-25T18:26:24.660 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:24.660 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:24.660 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:24.660 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:24.660 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:24.660 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:24.660 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:24.660 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:24.660 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:24.660 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:24.661 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.657+0000 7f8629d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:24.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.671+0000 7f86227fc640 1 -- 172.21.15.99:0/551684267 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86140017f0 con 0x7f86240a98f0 2024-03-25T18:26:24.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.671+0000 7f86227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:24.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.671+0000 7f86227fc640 1 -- 172.21.15.99:0/551684267 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8614001c00 con 0x7f86240a98f0 2024-03-25T18:26:24.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.671+0000 7f8629d32640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:24.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.671+0000 7f8629d32640 4 set_mon_vals no callback set 2024-03-25T18:26:24.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.671+0000 7f86227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:24.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.671+0000 7f86227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:24.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.671+0000 7f86227fc640 10 monclient: dump: 2024-03-25T18:26:24.672 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:24.672 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:24.672 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:24.672 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:24.672 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:24.672 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:24.673 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:24.673 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:24.673 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:24.673 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:24.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.671+0000 7f8629d32640 10 monclient: shutdown 2024-03-25T18:26:24.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.671+0000 7f8629d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:24.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.671+0000 7f8629d32640 1 -- 172.21.15.99:0/551684267 >> v1:172.21.15.190:6789/0 conn(0x7f86240a98f0 legacy=0x7f86240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:24.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.671+0000 7f8629d32640 1 -- 172.21.15.99:0/551684267 shutdown_connections 2024-03-25T18:26:24.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.671+0000 7f8629d32640 1 -- 172.21.15.99:0/551684267 >> 172.21.15.99:0/551684267 conn(0x7f86240b7ea0 msgr2=0x7f86240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:24.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.672+0000 7f8629d32640 1 -- 172.21.15.99:0/551684267 shutdown_connections 2024-03-25T18:26:24.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.672+0000 7f8629d32640 1 -- 172.21.15.99:0/551684267 wait complete. 2024-03-25T18:26:24.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.672+0000 7f8629d32640 10 monclient: build_initial_monmap 2024-03-25T18:26:24.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.672+0000 7f8629d32640 10 monclient: monmap: 2024-03-25T18:26:24.675 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:24.675 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:24.675 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:26:24.675 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:26:24.675 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:24.675 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:26:24.675 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:24.676 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:26:24.676 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:26:24.676 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:24.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.673+0000 7f8629d32640 1 librados: starting msgr at 2024-03-25T18:26:24.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.673+0000 7f8629d32640 1 librados: starting objecter 2024-03-25T18:26:24.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.673+0000 7f8629d32640 1 Processor -- start 2024-03-25T18:26:24.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.673+0000 7f8629d32640 1 -- start start 2024-03-25T18:26:24.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.673+0000 7f8629d32640 1 librados: setting wanted keys 2024-03-25T18:26:24.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.673+0000 7f8629d32640 1 librados: calling monclient init 2024-03-25T18:26:24.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.673+0000 7f8629d32640 10 monclient: init 2024-03-25T18:26:24.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.674+0000 7f8629d32640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:24.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.674+0000 7f8629d32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:24.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.674+0000 7f8629d32640 10 monclient(hunting): picked mon.noname-a con 0x7f86240a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:24.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.674+0000 7f8629d32640 10 monclient(hunting): picked mon.noname-b con 0x7f862413f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:24.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.674+0000 7f8629d32640 10 monclient(hunting): picked mon.noname-c con 0x7f86241430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:24.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.674+0000 7f8629d32640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8624138eb0 con 0x7f86240a98f0 2024-03-25T18:26:24.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.674+0000 7f8629d32640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8624139120 con 0x7f862413f710 2024-03-25T18:26:24.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.674+0000 7f8629d32640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8624132c50 con 0x7f86241430c0 2024-03-25T18:26:24.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.674+0000 7f8629d32640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:24.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.674+0000 7f8629d32640 10 monclient(hunting): authenticate will time out at 7402.842709s 2024-03-25T18:26:24.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.680+0000 7f8623fff640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f86241430c0 0x7f8624132540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:54834/0 (socket says 172.21.15.99:54834) 2024-03-25T18:26:24.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.680+0000 7f8623fff640 1 -- 172.21.15.99:0/3732307753 learned_addr learned my addr 172.21.15.99:0/3732307753 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:24.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.680+0000 7f860bfff640 10 client.?.objecter ms_handle_connect 0x7f86241430c0 2024-03-25T18:26:24.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.681+0000 7f860bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:24.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.688+0000 7f860bfff640 10 client.?.objecter ms_handle_connect 0x7f862413f710 2024-03-25T18:26:24.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.688+0000 7f860bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:24.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.688+0000 7f860bfff640 10 client.?.objecter ms_handle_connect 0x7f86240a98f0 2024-03-25T18:26:24.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.688+0000 7f860bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:24.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.699+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940782548 0 0) 0x7f8624138eb0 con 0x7f86240a98f0 2024-03-25T18:26:24.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.699+0000 7f860bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:24.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.699+0000 7f860bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:24.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.699+0000 7f860bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:24.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.699+0000 7f860bfff640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:26:24.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.699+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8610004eb0 con 0x7f86240a98f0 2024-03-25T18:26:24.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.701+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278227616 0 0) 0x7f8624132c50 con 0x7f86241430c0 2024-03-25T18:26:24.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.701+0000 7f860bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:24.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.701+0000 7f860bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:24.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.701+0000 7f860bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:24.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.701+0000 7f860bfff640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:26:24.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.701+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8624138eb0 con 0x7f86241430c0 2024-03-25T18:26:24.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.705+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198504477 0 0) 0x7f8624139120 con 0x7f862413f710 2024-03-25T18:26:24.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.705+0000 7f860bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:24.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.705+0000 7f860bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:24.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.705+0000 7f860bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:24.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.705+0000 7f860bfff640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:26:24.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.705+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8624132c50 con 0x7f862413f710 2024-03-25T18:26:24.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.710+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3201645548 0 0) 0x7f8624138eb0 con 0x7f86241430c0 2024-03-25T18:26:24.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.710+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8624139120 con 0x7f86241430c0 2024-03-25T18:26:24.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.710+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f861c001bd0 con 0x7f86241430c0 2024-03-25T18:26:24.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.710+0000 7f860bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:24.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.710+0000 7f860bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:24.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.710+0000 7f860bfff640 10 monclient(hunting): dump: 2024-03-25T18:26:24.711 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:24.711 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:24.712 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:24.712 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:24.712 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:24.712 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:24.712 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:24.712 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:24.712 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:24.712 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:24.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.710+0000 7f860bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:24.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.716+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1979076931 0 0) 0x7f8624132c50 con 0x7f862413f710 2024-03-25T18:26:24.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.716+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8624138eb0 con 0x7f862413f710 2024-03-25T18:26:24.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.716+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8618001930 con 0x7f862413f710 2024-03-25T18:26:24.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.716+0000 7f860bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:24.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.716+0000 7f860bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:24.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.716+0000 7f860bfff640 10 monclient(hunting): dump: 2024-03-25T18:26:24.718 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:24.718 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:24.718 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:24.718 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:24.718 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:24.718 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:24.718 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:24.718 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:24.718 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:24.718 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:24.720 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.719+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 972220138 0 0) 0x7f8610004eb0 con 0x7f86240a98f0 2024-03-25T18:26:24.720 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.719+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8624132c50 con 0x7f86240a98f0 2024-03-25T18:26:24.721 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.719+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8614001bc0 con 0x7f86240a98f0 2024-03-25T18:26:24.721 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.719+0000 7f860bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:24.721 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.719+0000 7f860bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:24.721 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.719+0000 7f860bfff640 10 monclient(hunting): dump: 2024-03-25T18:26:24.721 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:24.721 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:24.721 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:24.721 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:24.721 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:24.721 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:24.721 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:24.721 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:24.721 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:24.721 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:24.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 861374036 0 0) 0x7f8624139120 con 0x7f86241430c0 2024-03-25T18:26:24.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 >> v1:172.21.15.190:6789/0 conn(0x7f862413f710 legacy=0x7f8624131e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:24.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 >> v1:172.21.15.99:6789/0 conn(0x7f86240a98f0 legacy=0x7f8624137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:24.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f860bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:24.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f860bfff640 1 monclient: found mon.c 2024-03-25T18:26:24.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f860bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:24.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f860bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:26:24.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8624139400 con 0x7f86241430c0 2024-03-25T18:26:24.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f860bfff640 10 monclient: _finish_auth 0 2024-03-25T18:26:24.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f860bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:24.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f860bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:24.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f8629d32640 5 monclient: authenticate success, global_id 4277 2024-03-25T18:26:24.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f8629d32640 10 monclient: _renew_subs 2024-03-25T18:26:24.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f8629d32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:26:24.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f8629d32640 1 -- 172.21.15.99:0/3732307753 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8624137640 con 0x7f86241430c0 2024-03-25T18:26:24.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f8629d32640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:24.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f8629d32640 10 monclient: _renew_subs 2024-03-25T18:26:24.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f8629d32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:26:24.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f8629d32640 1 -- 172.21.15.99:0/3732307753 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86241382a0 con 0x7f86241430c0 2024-03-25T18:26:24.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.723+0000 7f8629d32640 1 librados: init done 2024-03-25T18:26:24.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.724+0000 7f8629d32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:24.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.724+0000 7f8629d32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:24.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.724+0000 7f8629d32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:26:24.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:24.724+0000 7f8629d32640 1 -- 172.21.15.99:0/3732307753 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86241374b0 con 0x7f86241430c0 2024-03-25T18:26:25.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.034+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f861c0020a0 con 0x7f86241430c0 2024-03-25T18:26:25.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.034+0000 7f860bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:25.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.034+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f861c003b30 con 0x7f86241430c0 2024-03-25T18:26:25.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.034+0000 7f860bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:25.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.034+0000 7f860bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:25.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.034+0000 7f860bfff640 10 monclient: dump: 2024-03-25T18:26:25.035 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:25.035 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:25.035 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:25.036 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:25.036 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:25.036 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:25.036 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:25.036 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:25.036 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:25.036 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:25.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.034+0000 7f8621ffb640 4 set_mon_vals no callback set 2024-03-25T18:26:25.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.034+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f861c003d50 con 0x7f86241430c0 2024-03-25T18:26:25.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.035+0000 7f86237fe640 10 monclient: get_auth_request con 0x7f861006e060 auth_method 0 2024-03-25T18:26:25.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.035+0000 7f860bfff640 10 client.4277.objecter ms_handle_connect 0x7f861006e060 2024-03-25T18:26:25.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.041+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f861c054a70 con 0x7f86241430c0 2024-03-25T18:26:25.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.041+0000 7f860bfff640 10 client.4277.objecter ms_dispatch 0x7f8624000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:25.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.041+0000 7f860bfff640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:25.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.041+0000 7f860bfff640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:25.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.041+0000 7f860bfff640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-25T18:26:25.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.041+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f861c0503b0 con 0x7f86241430c0 2024-03-25T18:26:25.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.041+0000 7f860bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:25.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.041+0000 7f860bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:25.266 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.264+0000 7f8629d32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:25.266 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.264+0000 7f8629d32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:25.266 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.264+0000 7f8629d32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:26:25.266 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.264+0000 7f8629d32640 1 -- 172.21.15.99:0/3732307753 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86241377d0 con 0x7f86241430c0 2024-03-25T18:26:25.293 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.292+0000 7f860bfff640 1 -- 172.21.15.99:0/3732307753 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f861c0524b0 con 0x7f86241430c0 2024-03-25T18:26:25.293 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.292+0000 7f860bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:25.293 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.292+0000 7f860bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:25.294 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:26:25.294 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:25.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.294+0000 7f8629d32640 10 librados: watch_flush enter 2024-03-25T18:26:25.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.294+0000 7f8629d32640 10 librados: watch_flush exit 2024-03-25T18:26:25.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.294+0000 7f8629d32640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-25T18:26:25.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.294+0000 7f8629d32640 10 client.4277.objecter successfully canceled tick 2024-03-25T18:26:25.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.294+0000 7f8629d32640 1 -- 172.21.15.99:0/3732307753 >> v1:172.21.15.190:6800/4220409699 conn(0x7f861006e060 legacy=0x7f8610070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.294+0000 7f8629d32640 10 monclient: shutdown 2024-03-25T18:26:25.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.294+0000 7f8629d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:25.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.294+0000 7f8629d32640 1 -- 172.21.15.99:0/3732307753 >> v1:172.21.15.99:6790/0 conn(0x7f86241430c0 legacy=0x7f8624132540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.297 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.294+0000 7f8629d32640 1 -- 172.21.15.99:0/3732307753 shutdown_connections 2024-03-25T18:26:25.297 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.294+0000 7f8629d32640 1 -- 172.21.15.99:0/3732307753 >> 172.21.15.99:0/3732307753 conn(0x7f86240b7ea0 msgr2=0x7f8624074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:25.297 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.295+0000 7f8629d32640 1 -- 172.21.15.99:0/3732307753 shutdown_connections 2024-03-25T18:26:25.297 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.295+0000 7f8629d32640 1 -- 172.21.15.99:0/3732307753 wait complete. 2024-03-25T18:26:25.297 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:25.295+0000 7f8629d32640 1 librados: shutdown 2024-03-25T18:26:25.308 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:32.116 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:32.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.229+0000 7f6a04a86640 10 monclient: get_monmap_and_config 2024-03-25T18:26:32.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.229+0000 7f6a04a86640 10 monclient: build_initial_monmap 2024-03-25T18:26:32.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.229+0000 7f6a04a86640 10 monclient: monmap: 2024-03-25T18:26:32.231 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:32.231 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:32.231 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:26:32.230648+0000 2024-03-25T18:26:32.231 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:26:32.230648+0000 2024-03-25T18:26:32.231 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:32.231 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:32.231 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:32.231 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:26:32.231 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:26:32.231 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:32.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.230+0000 7f6a04a86640 1 Processor -- start 2024-03-25T18:26:32.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.230+0000 7f6a04a86640 1 -- start start 2024-03-25T18:26:32.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.230+0000 7f6a04a86640 10 monclient: init 2024-03-25T18:26:32.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.231+0000 7f6a04a86640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:32.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.231+0000 7f6a04a86640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:32.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.231+0000 7f6a04a86640 10 monclient(hunting): picked mon.noname-a con 0x7f6a000a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:32.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.231+0000 7f6a04a86640 10 monclient(hunting): picked mon.noname-b con 0x7f6a0013f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:32.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.231+0000 7f6a04a86640 10 monclient(hunting): picked mon.noname-c con 0x7f6a001430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:32.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.231+0000 7f6a04a86640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a00142520 con 0x7f6a000a98f0 2024-03-25T18:26:32.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.231+0000 7f6a04a86640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a000b28a0 con 0x7f6a001430c0 2024-03-25T18:26:32.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.231+0000 7f6a04a86640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a000b2a30 con 0x7f6a0013f710 2024-03-25T18:26:32.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.231+0000 7f6a04a86640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:32.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.231+0000 7f6a04a86640 10 monclient(hunting): authenticate will time out at 7410.399447s 2024-03-25T18:26:32.242 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.241+0000 7f69fe59c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f6a000a98f0 0x7f6a000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34892/0 (socket says 172.21.15.99:34892) 2024-03-25T18:26:32.242 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.241+0000 7f69fe59c640 1 -- 172.21.15.99:0/667730434 learned_addr learned my addr 172.21.15.99:0/667730434 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:32.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.258+0000 7f69fdd9b640 1 -- 172.21.15.99:0/667730434 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556627475 0 0) 0x7f6a000b28a0 con 0x7f6a001430c0 2024-03-25T18:26:32.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.258+0000 7f69fdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:32.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.258+0000 7f69fdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:32.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.258+0000 7f69fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:32.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.258+0000 7f69fdd9b640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:26:32.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.258+0000 7f69fdd9b640 1 -- 172.21.15.99:0/667730434 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69ec003520 con 0x7f6a001430c0 2024-03-25T18:26:32.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.264+0000 7f69fdd9b640 1 -- 172.21.15.99:0/667730434 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987556029 0 0) 0x7f6a000b2a30 con 0x7f6a0013f710 2024-03-25T18:26:32.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.265+0000 7f69fdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:32.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.265+0000 7f69fdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:32.266 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.265+0000 7f69fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:32.266 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.265+0000 7f69fdd9b640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:26:32.266 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.265+0000 7f69fdd9b640 1 -- 172.21.15.99:0/667730434 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a000b28a0 con 0x7f6a0013f710 2024-03-25T18:26:32.268 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 1 -- 172.21.15.99:0/667730434 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491143271 0 0) 0x7f6a00142520 con 0x7f6a000a98f0 2024-03-25T18:26:32.268 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:32.268 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:32.268 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 1 -- 172.21.15.99:0/667730434 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a000b2a30 con 0x7f6a000a98f0 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 1 -- 172.21.15.99:0/667730434 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3983886624 0 0) 0x7f69ec003520 con 0x7f6a001430c0 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 1 -- 172.21.15.99:0/667730434 >> v1:172.21.15.99:6790/0 conn(0x7f6a0013f710 legacy=0x7f6a001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 1 -- 172.21.15.99:0/667730434 >> v1:172.21.15.99:6789/0 conn(0x7f6a000a98f0 legacy=0x7f6a000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 1 monclient: found mon.noname-c 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 1 -- 172.21.15.99:0/667730434 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a000b44c0 con 0x7f6a001430c0 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 10 monclient: _finish_auth 0 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 1 -- 172.21.15.99:0/667730434 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f69f8001a10 con 0x7f6a001430c0 2024-03-25T18:26:32.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:32.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:32.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f69fdd9b640 10 monclient: dump: 2024-03-25T18:26:32.270 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:32.270 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:32.270 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:32.270 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:32.270 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:32.270 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:32.270 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:32.270 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:32.270 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:32.270 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:32.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f6a04a86640 5 monclient: authenticate success, global_id 4279 2024-03-25T18:26:32.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.266+0000 7f6a04a86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:32.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.286+0000 7f69fdd9b640 1 -- 172.21.15.99:0/667730434 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69f8001b70 con 0x7f6a001430c0 2024-03-25T18:26:32.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.286+0000 7f69fdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:32.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.286+0000 7f69fdd9b640 1 -- 172.21.15.99:0/667730434 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f69f8001f80 con 0x7f6a001430c0 2024-03-25T18:26:32.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.286+0000 7f69fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:32.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.286+0000 7f69fdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:32.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.286+0000 7f69fdd9b640 10 monclient: dump: 2024-03-25T18:26:32.287 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:32.287 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:32.287 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:32.287 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:32.287 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:32.288 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:32.288 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:32.288 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:32.288 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:32.288 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:32.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.286+0000 7f6a04a86640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:32.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.286+0000 7f6a04a86640 4 set_mon_vals no callback set 2024-03-25T18:26:32.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.287+0000 7f6a04a86640 10 monclient: shutdown 2024-03-25T18:26:32.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.287+0000 7f6a04a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:32.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.287+0000 7f6a04a86640 1 -- 172.21.15.99:0/667730434 >> v1:172.21.15.190:6789/0 conn(0x7f6a001430c0 legacy=0x7f6a0014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:32.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.287+0000 7f6a04a86640 1 -- 172.21.15.99:0/667730434 shutdown_connections 2024-03-25T18:26:32.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.287+0000 7f6a04a86640 1 -- 172.21.15.99:0/667730434 >> 172.21.15.99:0/667730434 conn(0x7f6a000b7ea0 msgr2=0x7f6a000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:32.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.287+0000 7f6a04a86640 1 -- 172.21.15.99:0/667730434 shutdown_connections 2024-03-25T18:26:32.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.287+0000 7f6a04a86640 1 -- 172.21.15.99:0/667730434 wait complete. 2024-03-25T18:26:32.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.288+0000 7f6a04a86640 10 monclient: build_initial_monmap 2024-03-25T18:26:32.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.288+0000 7f6a04a86640 10 monclient: monmap: 2024-03-25T18:26:32.290 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:32.290 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:32.290 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:26:32.290 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:26:32.290 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:32.290 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:26:32.290 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:32.291 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:26:32.291 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:26:32.291 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:32.291 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.288+0000 7f6a04a86640 1 librados: starting msgr at 2024-03-25T18:26:32.291 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.288+0000 7f6a04a86640 1 librados: starting objecter 2024-03-25T18:26:32.291 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.288+0000 7f6a04a86640 1 Processor -- start 2024-03-25T18:26:32.291 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.289+0000 7f6a04a86640 1 -- start start 2024-03-25T18:26:32.291 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.289+0000 7f6a04a86640 1 librados: setting wanted keys 2024-03-25T18:26:32.291 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.289+0000 7f6a04a86640 1 librados: calling monclient init 2024-03-25T18:26:32.291 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.289+0000 7f6a04a86640 10 monclient: init 2024-03-25T18:26:32.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.289+0000 7f6a04a86640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:32.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.289+0000 7f6a04a86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:32.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.289+0000 7f6a04a86640 10 monclient(hunting): picked mon.noname-a con 0x7f6a000a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:32.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.289+0000 7f6a04a86640 10 monclient(hunting): picked mon.noname-b con 0x7f6a0013f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:32.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.289+0000 7f6a04a86640 10 monclient(hunting): picked mon.noname-c con 0x7f6a001430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:32.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.289+0000 7f6a04a86640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a00134d10 con 0x7f6a000a98f0 2024-03-25T18:26:32.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.289+0000 7f6a04a86640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a001351a0 con 0x7f6a0013f710 2024-03-25T18:26:32.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.289+0000 7f6a04a86640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a00133c00 con 0x7f6a001430c0 2024-03-25T18:26:32.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.289+0000 7f6a04a86640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:32.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.289+0000 7f6a04a86640 10 monclient(hunting): authenticate will time out at 7410.457470s 2024-03-25T18:26:32.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.295+0000 7f69fed9d640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f6a001430c0 0x7f6a001e1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:59598/0 (socket says 172.21.15.99:59598) 2024-03-25T18:26:32.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.295+0000 7f69fed9d640 1 -- 172.21.15.99:0/1853930492 learned_addr learned my addr 172.21.15.99:0/1853930492 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:32.299 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.298+0000 7f69f6ffd640 10 client.?.objecter ms_handle_connect 0x7f6a001430c0 2024-03-25T18:26:32.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.298+0000 7f69f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:32.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.299+0000 7f69f6ffd640 10 client.?.objecter ms_handle_connect 0x7f6a000a98f0 2024-03-25T18:26:32.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.299+0000 7f69f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:32.307 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.306+0000 7f69f6ffd640 10 client.?.objecter ms_handle_connect 0x7f6a0013f710 2024-03-25T18:26:32.307 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.306+0000 7f69f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:32.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.317+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075878022 0 0) 0x7f6a001351a0 con 0x7f6a0013f710 2024-03-25T18:26:32.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.317+0000 7f69f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:32.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.317+0000 7f69f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:32.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.317+0000 7f69f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:32.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.317+0000 7f69f6ffd640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:26:32.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.317+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e0004eb0 con 0x7f6a0013f710 2024-03-25T18:26:32.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.317+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381936170 0 0) 0x7f6a00134d10 con 0x7f6a000a98f0 2024-03-25T18:26:32.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.317+0000 7f69f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:32.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.317+0000 7f69f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:32.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.317+0000 7f69f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:32.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.317+0000 7f69f6ffd640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:26:32.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.317+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a001351a0 con 0x7f6a000a98f0 2024-03-25T18:26:32.321 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.320+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610865680 0 0) 0x7f6a00133c00 con 0x7f6a001430c0 2024-03-25T18:26:32.321 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.320+0000 7f69f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:32.321 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.320+0000 7f69f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:32.321 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.320+0000 7f69f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:32.321 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.320+0000 7f69f6ffd640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:26:32.321 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.320+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a00134d10 con 0x7f6a001430c0 2024-03-25T18:26:32.330 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.329+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2553356828 0 0) 0x7f6a00134d10 con 0x7f6a001430c0 2024-03-25T18:26:32.330 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.329+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a00133c00 con 0x7f6a001430c0 2024-03-25T18:26:32.330 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.329+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f69f8001e80 con 0x7f6a001430c0 2024-03-25T18:26:32.330 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.329+0000 7f69f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:32.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.329+0000 7f69f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:32.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.329+0000 7f69f6ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:32.331 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:32.331 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:32.331 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:32.331 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:32.331 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:32.331 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:32.331 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:32.331 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:32.331 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:32.331 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:32.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.329+0000 7f69f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:32.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.335+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1321450912 0 0) 0x7f69e0004eb0 con 0x7f6a0013f710 2024-03-25T18:26:32.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.335+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a00134d10 con 0x7f6a0013f710 2024-03-25T18:26:32.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.335+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f69e8001930 con 0x7f6a0013f710 2024-03-25T18:26:32.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.335+0000 7f69f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:32.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.335+0000 7f69f6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:32.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.335+0000 7f69f6ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:32.337 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:32.337 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:32.337 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:32.337 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:32.337 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:32.337 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:32.337 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:32.337 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:32.338 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:32.338 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:32.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.336+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2822078037 0 0) 0x7f6a001351a0 con 0x7f6a000a98f0 2024-03-25T18:26:32.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.336+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69e0004eb0 con 0x7f6a000a98f0 2024-03-25T18:26:32.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.336+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f69f0001930 con 0x7f6a000a98f0 2024-03-25T18:26:32.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.336+0000 7f69f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:32.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.336+0000 7f69f6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:32.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.336+0000 7f69f6ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:32.339 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:32.339 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:32.339 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:32.339 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:32.340 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:32.340 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:32.340 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:32.340 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:32.340 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:32.340 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:32.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 746195520 0 0) 0x7f69e0004eb0 con 0x7f6a000a98f0 2024-03-25T18:26:32.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 >> v1:172.21.15.99:6790/0 conn(0x7f6a001430c0 legacy=0x7f6a001e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:32.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 >> v1:172.21.15.190:6789/0 conn(0x7f6a0013f710 legacy=0x7f6a001db310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:32.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f69f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:32.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f69f6ffd640 1 monclient: found mon.a 2024-03-25T18:26:32.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f69f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:32.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f69f6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:26:32.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a00133d90 con 0x7f6a000a98f0 2024-03-25T18:26:32.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f69f6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:32.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f69f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:32.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f69f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:32.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f6a04a86640 5 monclient: authenticate success, global_id 4272 2024-03-25T18:26:32.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f6a04a86640 10 monclient: _renew_subs 2024-03-25T18:26:32.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f6a04a86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:26:32.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f6a04a86640 1 -- 172.21.15.99:0/1853930492 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a00132010 con 0x7f6a000a98f0 2024-03-25T18:26:32.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f6a04a86640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:32.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f6a04a86640 10 monclient: _renew_subs 2024-03-25T18:26:32.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f6a04a86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:26:32.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f6a04a86640 1 -- 172.21.15.99:0/1853930492 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a00135630 con 0x7f6a000a98f0 2024-03-25T18:26:32.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.347+0000 7f6a04a86640 1 librados: init done 2024-03-25T18:26:32.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.348+0000 7f6a04a86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:32.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.348+0000 7f6a04a86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:32.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.348+0000 7f6a04a86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:26:32.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.348+0000 7f6a04a86640 1 -- 172.21.15.99:0/1853930492 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a000627b0 con 0x7f6a000a98f0 2024-03-25T18:26:32.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.641+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69f00047e0 con 0x7f6a000a98f0 2024-03-25T18:26:32.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.641+0000 7f69f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:32.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.641+0000 7f69fd59a640 4 set_mon_vals no callback set 2024-03-25T18:26:32.654 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.653+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f69f0004af0 con 0x7f6a000a98f0 2024-03-25T18:26:32.654 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.653+0000 7f69f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:32.654 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.653+0000 7f69f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:32.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.653+0000 7f69f6ffd640 10 monclient: dump: 2024-03-25T18:26:32.655 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:32.655 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:32.655 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:32.655 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:32.655 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:32.655 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:32.655 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:32.655 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:32.655 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:32.655 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:32.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.654+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f69f001aac0 con 0x7f6a000a98f0 2024-03-25T18:26:32.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.655+0000 7f69f7fff640 10 monclient: get_auth_request con 0x7f69e006e0c0 auth_method 0 2024-03-25T18:26:32.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.655+0000 7f69f6ffd640 10 client.4272.objecter ms_handle_connect 0x7f69e006e0c0 2024-03-25T18:26:32.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.664+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f69f00547b0 con 0x7f6a000a98f0 2024-03-25T18:26:32.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.664+0000 7f69f6ffd640 10 client.4272.objecter ms_dispatch 0x7f6a00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:32.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.664+0000 7f69f6ffd640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:32.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.664+0000 7f69f6ffd640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:32.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.664+0000 7f69f6ffd640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-25T18:26:32.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.664+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69f00500f0 con 0x7f6a000a98f0 2024-03-25T18:26:32.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.664+0000 7f69f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:32.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.664+0000 7f69f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:32.911 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.910+0000 7f6a04a86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:32.911 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.910+0000 7f6a04a86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:32.912 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.910+0000 7f6a04a86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:26:32.912 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.910+0000 7f6a04a86640 1 -- 172.21.15.99:0/1853930492 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a00134930 con 0x7f6a000a98f0 2024-03-25T18:26:32.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.932+0000 7f69f6ffd640 1 -- 172.21.15.99:0/1853930492 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f69f00521f0 con 0x7f6a000a98f0 2024-03-25T18:26:32.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.932+0000 7f69f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:32.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.932+0000 7f69f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:32.934 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:26:32.934 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:32.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.935+0000 7f6a04a86640 10 librados: watch_flush enter 2024-03-25T18:26:32.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.935+0000 7f6a04a86640 10 librados: watch_flush exit 2024-03-25T18:26:32.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.935+0000 7f6a04a86640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-25T18:26:32.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.935+0000 7f6a04a86640 10 client.4272.objecter successfully canceled tick 2024-03-25T18:26:32.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.935+0000 7f6a04a86640 1 -- 172.21.15.99:0/1853930492 >> v1:172.21.15.190:6800/4220409699 conn(0x7f69e006e0c0 legacy=0x7f69e0070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:32.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.935+0000 7f6a04a86640 10 monclient: shutdown 2024-03-25T18:26:32.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.935+0000 7f6a04a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:32.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.935+0000 7f6a04a86640 1 -- 172.21.15.99:0/1853930492 >> v1:172.21.15.99:6789/0 conn(0x7f6a000a98f0 legacy=0x7f6a001dac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:32.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.935+0000 7f6a04a86640 1 -- 172.21.15.99:0/1853930492 shutdown_connections 2024-03-25T18:26:32.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.935+0000 7f6a04a86640 1 -- 172.21.15.99:0/1853930492 >> 172.21.15.99:0/1853930492 conn(0x7f6a000b7ea0 msgr2=0x7f6a0013d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:32.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.935+0000 7f6a04a86640 1 -- 172.21.15.99:0/1853930492 shutdown_connections 2024-03-25T18:26:32.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.935+0000 7f6a04a86640 1 -- 172.21.15.99:0/1853930492 wait complete. 2024-03-25T18:26:32.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:32.935+0000 7f6a04a86640 1 librados: shutdown 2024-03-25T18:26:32.948 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:33.569 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:01:34.162 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.622 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:01:34.214 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.622 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:01:34.214 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.641 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:01:34.233 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.696 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:01:34.289 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.700 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:34.259 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.704 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:34.263 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.708 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:01:34.301 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.756 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:34.315 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:39.755 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:39.868 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.867+0000 7fa92efc6640 10 monclient: get_monmap_and_config 2024-03-25T18:26:39.868 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.867+0000 7fa92efc6640 10 monclient: build_initial_monmap 2024-03-25T18:26:39.868 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.867+0000 7fa92efc6640 10 monclient: monmap: 2024-03-25T18:26:39.868 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:39.868 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:39.868 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:26:39.868158+0000 2024-03-25T18:26:39.868 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:26:39.868158+0000 2024-03-25T18:26:39.868 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:39.868 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:39.868 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:39.869 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:26:39.869 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:26:39.869 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:39.870 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.867+0000 7fa92efc6640 1 Processor -- start 2024-03-25T18:26:39.870 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.867+0000 7fa92efc6640 1 -- start start 2024-03-25T18:26:39.870 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.867+0000 7fa92efc6640 10 monclient: init 2024-03-25T18:26:39.870 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.868+0000 7fa92efc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:39.871 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.868+0000 7fa92efc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:39.871 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.868+0000 7fa92efc6640 10 monclient(hunting): picked mon.noname-c con 0x7fa9280b10d0 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:39.871 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.868+0000 7fa92efc6640 10 monclient(hunting): picked mon.noname-a con 0x7fa9280b4360 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:39.871 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.868+0000 7fa92efc6640 10 monclient(hunting): picked mon.noname-b con 0x7fa9280a9470 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:39.871 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.868+0000 7fa92efc6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280a9ab0 con 0x7fa9280b4360 2024-03-25T18:26:39.871 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.868+0000 7fa92efc6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280b2400 con 0x7fa9280b10d0 2024-03-25T18:26:39.871 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.868+0000 7fa92efc6640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280aa2a0 con 0x7fa9280a9470 2024-03-25T18:26:39.871 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.868+0000 7fa92efc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:39.871 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.868+0000 7fa92efc6640 10 monclient(hunting): authenticate will time out at 7418.036427s 2024-03-25T18:26:39.871 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.869+0000 7fa92cd62640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fa9280b10d0 0x7fa9280b4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:60396/0 (socket says 172.21.15.99:60396) 2024-03-25T18:26:39.871 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.869+0000 7fa92cd62640 1 -- 172.21.15.99:0/1462854036 learned_addr learned my addr 172.21.15.99:0/1462854036 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:39.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.932+0000 7fa9277fe640 1 -- 172.21.15.99:0/1462854036 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749658281 0 0) 0x7fa9280aa2a0 con 0x7fa9280a9470 2024-03-25T18:26:39.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.932+0000 7fa9277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:39.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.932+0000 7fa9277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:39.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.932+0000 7fa9277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:39.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.932+0000 7fa9277fe640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:26:39.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.932+0000 7fa9277fe640 1 -- 172.21.15.99:0/1462854036 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa918003520 con 0x7fa9280a9470 2024-03-25T18:26:39.962 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 1 -- 172.21.15.99:0/1462854036 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1439857764 0 0) 0x7fa918003520 con 0x7fa9280a9470 2024-03-25T18:26:39.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 1 -- 172.21.15.99:0/1462854036 >> v1:172.21.15.190:6789/0 conn(0x7fa9280b10d0 legacy=0x7fa9280b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:39.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 1 -- 172.21.15.99:0/1462854036 >> v1:172.21.15.99:6789/0 conn(0x7fa9280b4360 legacy=0x7fa9280b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:39.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:39.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 1 monclient: found mon.noname-b 2024-03-25T18:26:39.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:39.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:26:39.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 1 -- 172.21.15.99:0/1462854036 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9280b14b0 con 0x7fa9280a9470 2024-03-25T18:26:39.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:39.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:39.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:39.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 1 -- 172.21.15.99:0/1462854036 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa920001a10 con 0x7fa9280a9470 2024-03-25T18:26:39.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:39.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:39.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa9277fe640 10 monclient: dump: 2024-03-25T18:26:39.964 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:39.964 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:39.964 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:39.964 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:39.964 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:39.964 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:39.964 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:39.964 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:39.964 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:39.964 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:39.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa92efc6640 5 monclient: authenticate success, global_id 4286 2024-03-25T18:26:39.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.961+0000 7fa92efc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:39.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.996+0000 7fa9277fe640 1 -- 172.21.15.99:0/1462854036 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa920001b70 con 0x7fa9280a9470 2024-03-25T18:26:39.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.996+0000 7fa9277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:39.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.996+0000 7fa9277fe640 1 -- 172.21.15.99:0/1462854036 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa920001fc0 con 0x7fa9280a9470 2024-03-25T18:26:39.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.996+0000 7fa92efc6640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:39.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.996+0000 7fa92efc6640 4 set_mon_vals no callback set 2024-03-25T18:26:39.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.996+0000 7fa9277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:39.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.996+0000 7fa9277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:39.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.996+0000 7fa9277fe640 10 monclient: dump: 2024-03-25T18:26:39.997 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:39.997 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:39.997 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:39.998 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:39.998 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:39.998 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:39.998 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:39.998 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:39.998 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:39.998 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:40.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.996+0000 7fa92efc6640 10 monclient: shutdown 2024-03-25T18:26:40.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.996+0000 7fa92efc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:40.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.996+0000 7fa92efc6640 1 -- 172.21.15.99:0/1462854036 >> v1:172.21.15.99:6790/0 conn(0x7fa9280a9470 legacy=0x7fa92814dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.997+0000 7fa92efc6640 1 -- 172.21.15.99:0/1462854036 shutdown_connections 2024-03-25T18:26:40.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.997+0000 7fa92efc6640 1 -- 172.21.15.99:0/1462854036 >> 172.21.15.99:0/1462854036 conn(0x7fa9280617a0 msgr2=0x7fa928061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.997+0000 7fa92efc6640 1 -- 172.21.15.99:0/1462854036 shutdown_connections 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.997+0000 7fa92efc6640 1 -- 172.21.15.99:0/1462854036 wait complete. 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.997+0000 7fa92efc6640 10 monclient: build_initial_monmap 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.997+0000 7fa92efc6640 10 monclient: monmap: 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.998+0000 7fa92efc6640 1 librados: starting msgr at 2024-03-25T18:26:40.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.998+0000 7fa92efc6640 1 librados: starting objecter 2024-03-25T18:26:40.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.998+0000 7fa92efc6640 1 Processor -- start 2024-03-25T18:26:40.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.998+0000 7fa92efc6640 1 -- start start 2024-03-25T18:26:40.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.998+0000 7fa92efc6640 1 librados: setting wanted keys 2024-03-25T18:26:40.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.998+0000 7fa92efc6640 1 librados: calling monclient init 2024-03-25T18:26:40.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.998+0000 7fa92efc6640 10 monclient: init 2024-03-25T18:26:40.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.999+0000 7fa92efc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:40.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.999+0000 7fa92efc6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:40.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.999+0000 7fa92efc6640 10 monclient(hunting): picked mon.noname-a con 0x7fa9280a9470 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:40.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.999+0000 7fa92efc6640 10 monclient(hunting): picked mon.noname-b con 0x7fa9280b10d0 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:40.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.999+0000 7fa92efc6640 10 monclient(hunting): picked mon.noname-c con 0x7fa9280b4360 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:40.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.999+0000 7fa92efc6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa928131250 con 0x7fa9280a9470 2024-03-25T18:26:40.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.999+0000 7fa92efc6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa928131530 con 0x7fa9280b10d0 2024-03-25T18:26:40.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.999+0000 7fa92efc6640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9281408a0 con 0x7fa9280b4360 2024-03-25T18:26:40.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.999+0000 7fa92efc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:40.003 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:39.999+0000 7fa92efc6640 10 monclient(hunting): authenticate will time out at 7418.167183s 2024-03-25T18:26:40.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.003+0000 7fa927fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fa9280b10d0 0x7fa928141f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:60446/0 (socket says 172.21.15.99:60446) 2024-03-25T18:26:40.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.003+0000 7fa927fff640 1 -- 172.21.15.99:0/1470755920 learned_addr learned my addr 172.21.15.99:0/1470755920 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:40.006 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.005+0000 7fa9257fa640 10 client.?.objecter ms_handle_connect 0x7fa9280b10d0 2024-03-25T18:26:40.006 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.005+0000 7fa9257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:40.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.013+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306058937 0 0) 0x7fa928131530 con 0x7fa9280b10d0 2024-03-25T18:26:40.014 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.013+0000 7fa9257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.014 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.013+0000 7fa9257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.014 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.013+0000 7fa9257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.014 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.013+0000 7fa9257fa640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:26:40.014 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.013+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa90c004eb0 con 0x7fa9280b10d0 2024-03-25T18:26:40.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.019+0000 7fa9257fa640 10 client.?.objecter ms_handle_connect 0x7fa9280b4360 2024-03-25T18:26:40.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.019+0000 7fa9257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:40.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.027+0000 7fa9257fa640 10 client.?.objecter ms_handle_connect 0x7fa9280a9470 2024-03-25T18:26:40.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.027+0000 7fa9257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:40.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.029+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1400918246 0 0) 0x7fa90c004eb0 con 0x7fa9280b10d0 2024-03-25T18:26:40.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.029+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa928131530 con 0x7fa9280b10d0 2024-03-25T18:26:40.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.029+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa9140018d0 con 0x7fa9280b10d0 2024-03-25T18:26:40.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.029+0000 7fa9257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.029+0000 7fa9257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:40.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.029+0000 7fa9257fa640 10 monclient(hunting): dump: 2024-03-25T18:26:40.031 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:40.031 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:40.031 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:40.031 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:40.031 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.031 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:40.031 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:40.031 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:40.031 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:40.031 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:40.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.029+0000 7fa9257fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:40.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.031+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905487551 0 0) 0x7fa9281408a0 con 0x7fa9280b4360 2024-03-25T18:26:40.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.031+0000 7fa9257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.031+0000 7fa9257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.033 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.031+0000 7fa9257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.033 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.031+0000 7fa9257fa640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:26:40.033 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.031+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa90c004eb0 con 0x7fa9280b4360 2024-03-25T18:26:40.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.041+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368967049 0 0) 0x7fa928131250 con 0x7fa9280a9470 2024-03-25T18:26:40.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.041+0000 7fa9257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.041+0000 7fa9257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.041+0000 7fa9257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.041+0000 7fa9257fa640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:26:40.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.041+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9281408a0 con 0x7fa9280a9470 2024-03-25T18:26:40.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2893472429 0 0) 0x7fa928131530 con 0x7fa9280b10d0 2024-03-25T18:26:40.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 >> v1:172.21.15.99:6790/0 conn(0x7fa9280b4360 legacy=0x7fa928142670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 >> v1:172.21.15.99:6789/0 conn(0x7fa9280a9470 legacy=0x7fa928131950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa9257fa640 10 monclient: _finish_hunting 0 2024-03-25T18:26:40.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa9257fa640 1 monclient: found mon.b 2024-03-25T18:26:40.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa9257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:40.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa9257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:40.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa928142d80 con 0x7fa9280b10d0 2024-03-25T18:26:40.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa9257fa640 10 monclient: _finish_auth 0 2024-03-25T18:26:40.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa9257fa640 10 monclient: _check_auth_tickets 2024-03-25T18:26:40.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa9257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:40.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa92efc6640 5 monclient: authenticate success, global_id 4288 2024-03-25T18:26:40.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa92efc6640 10 monclient: _renew_subs 2024-03-25T18:26:40.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa92efc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:40.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.046+0000 7fa92efc6640 1 -- 172.21.15.99:0/1470755920 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa928132060 con 0x7fa9280b10d0 2024-03-25T18:26:40.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.047+0000 7fa92efc6640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:40.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.047+0000 7fa92efc6640 10 monclient: _renew_subs 2024-03-25T18:26:40.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.047+0000 7fa92efc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:40.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.047+0000 7fa92efc6640 1 -- 172.21.15.99:0/1470755920 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9281353b0 con 0x7fa9280b10d0 2024-03-25T18:26:40.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.047+0000 7fa92efc6640 1 librados: init done 2024-03-25T18:26:40.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.048+0000 7fa92efc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:40.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.048+0000 7fa92efc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:40.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.048+0000 7fa92efc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:40.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.048+0000 7fa92efc6640 1 -- 172.21.15.99:0/1470755920 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa928140a30 con 0x7fa9280b10d0 2024-03-25T18:26:40.330 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.329+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa914003610 con 0x7fa9280b10d0 2024-03-25T18:26:40.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.329+0000 7fa9257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:40.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.329+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa914003a20 con 0x7fa9280b10d0 2024-03-25T18:26:40.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.329+0000 7fa9257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.329+0000 7fa9257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:40.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.329+0000 7fa9257fa640 10 monclient: dump: 2024-03-25T18:26:40.331 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:40.331 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:40.331 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:40.331 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:40.331 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.331 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:40.331 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:40.331 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:40.332 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:40.332 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:40.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.329+0000 7fa926ffd640 4 set_mon_vals no callback set 2024-03-25T18:26:40.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.330+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fa914003c40 con 0x7fa9280b10d0 2024-03-25T18:26:40.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.331+0000 7fa92cd62640 10 monclient: get_auth_request con 0x7fa90c06dea0 auth_method 0 2024-03-25T18:26:40.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.331+0000 7fa9257fa640 10 client.4288.objecter ms_handle_connect 0x7fa90c06dea0 2024-03-25T18:26:40.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.336+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fa9140547a0 con 0x7fa9280b10d0 2024-03-25T18:26:40.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.336+0000 7fa9257fa640 10 client.4288.objecter ms_dispatch 0x7fa928000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:40.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.336+0000 7fa9257fa640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:40.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.336+0000 7fa9257fa640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:40.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.336+0000 7fa9257fa640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T18:26:40.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.337+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9140502b0 con 0x7fa9280b10d0 2024-03-25T18:26:40.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.337+0000 7fa9257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:40.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.337+0000 7fa9257fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:40.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.582+0000 7fa92efc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:40.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.582+0000 7fa92efc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:40.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.582+0000 7fa92efc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:40.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.582+0000 7fa92efc6640 1 -- 172.21.15.99:0/1470755920 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9280627b0 con 0x7fa9280b10d0 2024-03-25T18:26:40.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.613+0000 7fa9257fa640 1 -- 172.21.15.99:0/1470755920 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fa9140521e0 con 0x7fa9280b10d0 2024-03-25T18:26:40.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.613+0000 7fa9257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:40.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.613+0000 7fa9257fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:40.614 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:26:40.615 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:40.616 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.615+0000 7fa92efc6640 10 librados: watch_flush enter 2024-03-25T18:26:40.616 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.615+0000 7fa92efc6640 10 librados: watch_flush exit 2024-03-25T18:26:40.616 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.615+0000 7fa92efc6640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T18:26:40.617 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.615+0000 7fa92efc6640 10 client.4288.objecter successfully canceled tick 2024-03-25T18:26:40.617 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.615+0000 7fa92efc6640 1 -- 172.21.15.99:0/1470755920 >> v1:172.21.15.190:6800/4220409699 conn(0x7fa90c06dea0 legacy=0x7fa90c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.617 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.615+0000 7fa92efc6640 10 monclient: shutdown 2024-03-25T18:26:40.617 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.615+0000 7fa92efc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:40.617 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.615+0000 7fa92efc6640 1 -- 172.21.15.99:0/1470755920 >> v1:172.21.15.190:6789/0 conn(0x7fa9280b10d0 legacy=0x7fa928141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.617 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.615+0000 7fa92efc6640 1 -- 172.21.15.99:0/1470755920 shutdown_connections 2024-03-25T18:26:40.617 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.615+0000 7fa92efc6640 1 -- 172.21.15.99:0/1470755920 >> 172.21.15.99:0/1470755920 conn(0x7fa9280617a0 msgr2=0x7fa928061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:40.617 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.615+0000 7fa92efc6640 1 -- 172.21.15.99:0/1470755920 shutdown_connections 2024-03-25T18:26:40.617 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.615+0000 7fa92efc6640 1 -- 172.21.15.99:0/1470755920 wait complete. 2024-03-25T18:26:40.617 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:40.615+0000 7fa92efc6640 1 librados: shutdown 2024-03-25T18:26:40.628 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:47.435 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:47.543 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.542+0000 7f79df2a8640 10 monclient: get_monmap_and_config 2024-03-25T18:26:47.543 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.542+0000 7f79df2a8640 10 monclient: build_initial_monmap 2024-03-25T18:26:47.543 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.542+0000 7f79df2a8640 10 monclient: monmap: 2024-03-25T18:26:47.544 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:47.544 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:47.544 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:26:47.543610+0000 2024-03-25T18:26:47.544 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:26:47.543610+0000 2024-03-25T18:26:47.544 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:47.544 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:47.544 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:47.544 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:26:47.544 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:26:47.544 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:47.545 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.543+0000 7f79df2a8640 1 Processor -- start 2024-03-25T18:26:47.545 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.543+0000 7f79df2a8640 1 -- start start 2024-03-25T18:26:47.545 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.543+0000 7f79df2a8640 10 monclient: init 2024-03-25T18:26:47.545 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.544+0000 7f79df2a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:47.545 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.544+0000 7f79df2a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:47.545 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.544+0000 7f79df2a8640 10 monclient(hunting): picked mon.noname-c con 0x7f79d80a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:47.545 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.544+0000 7f79df2a8640 10 monclient(hunting): picked mon.noname-b con 0x7f79d813f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:47.546 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.544+0000 7f79df2a8640 10 monclient(hunting): picked mon.noname-a con 0x7f79d81430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:47.546 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.544+0000 7f79df2a8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d8142520 con 0x7f79d81430c0 2024-03-25T18:26:47.546 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.544+0000 7f79df2a8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d80b28a0 con 0x7f79d80a98f0 2024-03-25T18:26:47.546 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.544+0000 7f79df2a8640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d80b2a30 con 0x7f79d813f710 2024-03-25T18:26:47.546 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.544+0000 7f79df2a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:47.546 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.544+0000 7f79df2a8640 10 monclient(hunting): authenticate will time out at 7425.712417s 2024-03-25T18:26:47.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.552+0000 7f79dd044640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f79d80a98f0 0x7f79d80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:48190/0 (socket says 172.21.15.99:48190) 2024-03-25T18:26:47.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.553+0000 7f79dd044640 1 -- 172.21.15.99:0/1753166923 learned_addr learned my addr 172.21.15.99:0/1753166923 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:47.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.568+0000 7f79cffff640 1 -- 172.21.15.99:0/1753166923 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876904688 0 0) 0x7f79d80b2a30 con 0x7f79d813f710 2024-03-25T18:26:47.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.568+0000 7f79cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:47.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.568+0000 7f79cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:47.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.568+0000 7f79cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:47.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.568+0000 7f79cffff640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:26:47.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.568+0000 7f79cffff640 1 -- 172.21.15.99:0/1753166923 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c4003520 con 0x7f79d813f710 2024-03-25T18:26:47.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.570+0000 7f79cffff640 1 -- 172.21.15.99:0/1753166923 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162204375 0 0) 0x7f79d80b28a0 con 0x7f79d80a98f0 2024-03-25T18:26:47.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.570+0000 7f79cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:47.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.570+0000 7f79cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:47.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.570+0000 7f79cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:47.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.570+0000 7f79cffff640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:26:47.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.570+0000 7f79cffff640 1 -- 172.21.15.99:0/1753166923 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79d80b2a30 con 0x7f79d80a98f0 2024-03-25T18:26:47.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 1 -- 172.21.15.99:0/1753166923 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2170990154 0 0) 0x7f79d80b2a30 con 0x7f79d80a98f0 2024-03-25T18:26:47.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 1 -- 172.21.15.99:0/1753166923 >> v1:172.21.15.99:6790/0 conn(0x7f79d813f710 legacy=0x7f79d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:47.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 1 -- 172.21.15.99:0/1753166923 >> v1:172.21.15.99:6789/0 conn(0x7f79d81430c0 legacy=0x7f79d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:47.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:47.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 1 monclient: found mon.noname-c 2024-03-25T18:26:47.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:47.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:26:47.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 1 -- 172.21.15.99:0/1753166923 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d80b44c0 con 0x7f79d80a98f0 2024-03-25T18:26:47.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 10 monclient: _finish_auth 0 2024-03-25T18:26:47.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:47.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:47.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 1 -- 172.21.15.99:0/1753166923 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f79d4001670 con 0x7f79d80a98f0 2024-03-25T18:26:47.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:47.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:47.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79cffff640 10 monclient: dump: 2024-03-25T18:26:47.584 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:47.584 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:47.584 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:47.584 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:47.584 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:47.584 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:47.584 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:47.584 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:47.584 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:47.585 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:47.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79df2a8640 5 monclient: authenticate success, global_id 4291 2024-03-25T18:26:47.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.581+0000 7f79df2a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:47.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.598+0000 7f79cffff640 1 -- 172.21.15.99:0/1753166923 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79d40017d0 con 0x7f79d80a98f0 2024-03-25T18:26:47.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.598+0000 7f79cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:47.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.598+0000 7f79df2a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:47.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.598+0000 7f79df2a8640 4 set_mon_vals no callback set 2024-03-25T18:26:47.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.599+0000 7f79df2a8640 10 monclient: shutdown 2024-03-25T18:26:47.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.599+0000 7f79df2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:47.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.599+0000 7f79df2a8640 1 -- 172.21.15.99:0/1753166923 >> v1:172.21.15.190:6789/0 conn(0x7f79d80a98f0 legacy=0x7f79d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:47.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.599+0000 7f79df2a8640 1 -- 172.21.15.99:0/1753166923 shutdown_connections 2024-03-25T18:26:47.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.599+0000 7f79df2a8640 1 -- 172.21.15.99:0/1753166923 >> 172.21.15.99:0/1753166923 conn(0x7f79d80b7ea0 msgr2=0x7f79d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:47.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.599+0000 7f79df2a8640 1 -- 172.21.15.99:0/1753166923 shutdown_connections 2024-03-25T18:26:47.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.599+0000 7f79df2a8640 1 -- 172.21.15.99:0/1753166923 wait complete. 2024-03-25T18:26:47.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.600+0000 7f79df2a8640 10 monclient: build_initial_monmap 2024-03-25T18:26:47.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.600+0000 7f79df2a8640 10 monclient: monmap: 2024-03-25T18:26:47.604 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:47.604 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:47.604 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:26:47.604 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:26:47.604 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:47.604 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:26:47.605 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:47.605 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:26:47.605 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:26:47.605 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:47.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.601+0000 7f79df2a8640 1 librados: starting msgr at 2024-03-25T18:26:47.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.601+0000 7f79df2a8640 1 librados: starting objecter 2024-03-25T18:26:47.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.601+0000 7f79df2a8640 1 Processor -- start 2024-03-25T18:26:47.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.601+0000 7f79df2a8640 1 -- start start 2024-03-25T18:26:47.606 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.601+0000 7f79df2a8640 1 librados: setting wanted keys 2024-03-25T18:26:47.606 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.601+0000 7f79df2a8640 1 librados: calling monclient init 2024-03-25T18:26:47.606 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.601+0000 7f79df2a8640 10 monclient: init 2024-03-25T18:26:47.606 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.601+0000 7f79df2a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:47.606 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.602+0000 7f79df2a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:47.606 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.602+0000 7f79df2a8640 10 monclient(hunting): picked mon.noname-c con 0x7f79d80a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:47.607 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.602+0000 7f79df2a8640 10 monclient(hunting): picked mon.noname-a con 0x7f79d813f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:47.607 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.602+0000 7f79df2a8640 10 monclient(hunting): picked mon.noname-b con 0x7f79d81430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:47.607 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.602+0000 7f79df2a8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d81dbac0 con 0x7f79d813f710 2024-03-25T18:26:47.607 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.602+0000 7f79df2a8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d81d51d0 con 0x7f79d81430c0 2024-03-25T18:26:47.607 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.602+0000 7f79df2a8640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d81d45a0 con 0x7f79d80a98f0 2024-03-25T18:26:47.607 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.602+0000 7f79df2a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:47.607 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.602+0000 7f79df2a8640 10 monclient(hunting): authenticate will time out at 7425.770071s 2024-03-25T18:26:47.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.608+0000 7f79dc843640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f79d813f710 0x7f79d81d4ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53076/0 (socket says 172.21.15.99:53076) 2024-03-25T18:26:47.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.608+0000 7f79dc843640 1 -- 172.21.15.99:0/1249879275 learned_addr learned my addr 172.21.15.99:0/1249879275 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:47.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.609+0000 7f79cdffb640 10 client.?.objecter ms_handle_connect 0x7f79d813f710 2024-03-25T18:26:47.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.609+0000 7f79cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:47.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.614+0000 7f79cdffb640 10 client.?.objecter ms_handle_connect 0x7f79d81430c0 2024-03-25T18:26:47.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.614+0000 7f79cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:47.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.620+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89587247 0 0) 0x7f79d81d51d0 con 0x7f79d81430c0 2024-03-25T18:26:47.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.620+0000 7f79cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:47.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.620+0000 7f79cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:47.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.620+0000 7f79cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:47.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.620+0000 7f79cdffb640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:26:47.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.620+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79bc004eb0 con 0x7f79d81430c0 2024-03-25T18:26:47.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.634+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1191757697 0 0) 0x7f79bc004eb0 con 0x7f79d81430c0 2024-03-25T18:26:47.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.634+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79d81d51d0 con 0x7f79d81430c0 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.634+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f79d0001b90 con 0x7f79d81430c0 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.634+0000 7f79cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.634+0000 7f79cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.634+0000 7f79cdffb640 10 monclient(hunting): dump: 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:47.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.634+0000 7f79cdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:47.640 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.639+0000 7f79cdffb640 10 client.?.objecter ms_handle_connect 0x7f79d80a98f0 2024-03-25T18:26:47.640 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.640+0000 7f79cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:47.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.641+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384241084 0 0) 0x7f79c4003520 con 0x7f79d813f710 2024-03-25T18:26:47.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.641+0000 7f79cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:47.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.641+0000 7f79cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:47.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.641+0000 7f79cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:47.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.641+0000 7f79cdffb640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:26:47.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.641+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79bc004eb0 con 0x7f79d813f710 2024-03-25T18:26:47.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3107723194 0 0) 0x7f79d81d51d0 con 0x7f79d81430c0 2024-03-25T18:26:47.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 >> v1:172.21.15.99:6790/0 conn(0x7f79d80a98f0 legacy=0x7f79d81db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:47.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 >> v1:172.21.15.99:6789/0 conn(0x7f79d813f710 legacy=0x7f79d81d4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:47.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79cdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:47.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79cdffb640 1 monclient: found mon.b 2024-03-25T18:26:47.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:47.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79cdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:47.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d81d41e0 con 0x7f79d81430c0 2024-03-25T18:26:47.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79cdffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:47.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79cdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:47.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:47.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79df2a8640 5 monclient: authenticate success, global_id 4294 2024-03-25T18:26:47.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79df2a8640 10 monclient: _renew_subs 2024-03-25T18:26:47.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79df2a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:47.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.646+0000 7f79df2a8640 1 -- 172.21.15.99:0/1249879275 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d8134690 con 0x7f79d81430c0 2024-03-25T18:26:47.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.647+0000 7f79df2a8640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:47.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.647+0000 7f79df2a8640 10 monclient: _renew_subs 2024-03-25T18:26:47.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.647+0000 7f79df2a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:47.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.647+0000 7f79df2a8640 1 -- 172.21.15.99:0/1249879275 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d81dade0 con 0x7f79d81430c0 2024-03-25T18:26:47.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.647+0000 7f79df2a8640 1 librados: init done 2024-03-25T18:26:47.650 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.648+0000 7f79df2a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:47.650 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.648+0000 7f79df2a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:47.650 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.648+0000 7f79df2a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:47.650 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.648+0000 7f79df2a8640 1 -- 172.21.15.99:0/1249879275 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d80b3ef0 con 0x7f79d81430c0 2024-03-25T18:26:47.938 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.936+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79d0003970 con 0x7f79d81430c0 2024-03-25T18:26:47.938 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.936+0000 7f79cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:47.938 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.936+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f79d0002100 con 0x7f79d81430c0 2024-03-25T18:26:47.938 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.936+0000 7f79cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:47.938 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.936+0000 7f79cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:47.939 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.936+0000 7f79cf7fe640 4 set_mon_vals no callback set 2024-03-25T18:26:47.939 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.936+0000 7f79cdffb640 10 monclient: dump: 2024-03-25T18:26:47.939 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:47.939 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:47.939 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:47.939 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:47.939 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:47.939 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:47.939 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:47.939 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:47.940 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:47.940 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:47.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.937+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f79d0003e00 con 0x7f79d81430c0 2024-03-25T18:26:47.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.938+0000 7f79dd044640 10 monclient: get_auth_request con 0x7f79bc06ddd0 auth_method 0 2024-03-25T18:26:47.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.938+0000 7f79cdffb640 10 client.4294.objecter ms_handle_connect 0x7f79bc06ddd0 2024-03-25T18:26:47.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.942+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f79d0053880 con 0x7f79d81430c0 2024-03-25T18:26:47.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.942+0000 7f79cdffb640 10 client.4294.objecter ms_dispatch 0x7f79d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:47.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.942+0000 7f79cdffb640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:47.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.942+0000 7f79cdffb640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:47.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.942+0000 7f79cdffb640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T18:26:47.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.943+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f79d004f390 con 0x7f79d81430c0 2024-03-25T18:26:47.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.943+0000 7f79cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:47.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:47.943+0000 7f79cdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:48.163 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.161+0000 7f79df2a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:48.163 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.161+0000 7f79df2a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:48.163 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.161+0000 7f79df2a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:26:48.163 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.161+0000 7f79df2a8640 1 -- 172.21.15.99:0/1249879275 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79d81d4730 con 0x7f79d81430c0 2024-03-25T18:26:48.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.190+0000 7f79cdffb640 1 -- 172.21.15.99:0/1249879275 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f79d00512c0 con 0x7f79d81430c0 2024-03-25T18:26:48.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.190+0000 7f79cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:48.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.190+0000 7f79cdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:48.192 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:26:48.192 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:48.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.192+0000 7f79df2a8640 10 librados: watch_flush enter 2024-03-25T18:26:48.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.192+0000 7f79df2a8640 10 librados: watch_flush exit 2024-03-25T18:26:48.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.192+0000 7f79df2a8640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T18:26:48.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.192+0000 7f79df2a8640 10 client.4294.objecter successfully canceled tick 2024-03-25T18:26:48.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.193+0000 7f79df2a8640 1 -- 172.21.15.99:0/1249879275 >> v1:172.21.15.190:6800/4220409699 conn(0x7f79bc06ddd0 legacy=0x7f79bc070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.193+0000 7f79df2a8640 10 monclient: shutdown 2024-03-25T18:26:48.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.193+0000 7f79df2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:48.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.193+0000 7f79df2a8640 1 -- 172.21.15.99:0/1249879275 >> v1:172.21.15.190:6789/0 conn(0x7f79d81430c0 legacy=0x7f79d8133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.193+0000 7f79df2a8640 1 -- 172.21.15.99:0/1249879275 shutdown_connections 2024-03-25T18:26:48.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.193+0000 7f79df2a8640 1 -- 172.21.15.99:0/1249879275 >> 172.21.15.99:0/1249879275 conn(0x7f79d80b7ea0 msgr2=0x7f79d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:48.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.193+0000 7f79df2a8640 1 -- 172.21.15.99:0/1249879275 shutdown_connections 2024-03-25T18:26:48.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.193+0000 7f79df2a8640 1 -- 172.21.15.99:0/1249879275 wait complete. 2024-03-25T18:26:48.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:48.193+0000 7f79df2a8640 1 librados: shutdown 2024-03-25T18:26:48.208 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:53.678 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:01:54.270 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.687 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:01:54.280 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.731 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:54.290 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.733 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:01:54.325 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.733 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:01:54.326 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.740 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:54.299 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.748 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:01:54.340 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.761 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:54.320 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:53.778 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:01:54.371 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:55.014 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:55.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.132+0000 7fb15ea60640 10 monclient: get_monmap_and_config 2024-03-25T18:26:55.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.132+0000 7fb15ea60640 10 monclient: build_initial_monmap 2024-03-25T18:26:55.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.132+0000 7fb15ea60640 10 monclient: monmap: 2024-03-25T18:26:55.134 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:55.134 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:55.134 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:26:55.133868+0000 2024-03-25T18:26:55.134 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:26:55.133868+0000 2024-03-25T18:26:55.134 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:55.134 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:55.134 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:55.134 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:26:55.135 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:26:55.135 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:55.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.133+0000 7fb15ea60640 1 Processor -- start 2024-03-25T18:26:55.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.133+0000 7fb15ea60640 1 -- start start 2024-03-25T18:26:55.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.133+0000 7fb15ea60640 10 monclient: init 2024-03-25T18:26:55.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.134+0000 7fb15ea60640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:55.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.134+0000 7fb15ea60640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:55.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.134+0000 7fb15ea60640 10 monclient(hunting): picked mon.noname-b con 0x7fb1580a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:55.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.134+0000 7fb15ea60640 10 monclient(hunting): picked mon.noname-a con 0x7fb15813f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:55.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.134+0000 7fb15ea60640 10 monclient(hunting): picked mon.noname-c con 0x7fb1581430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:55.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.134+0000 7fb15ea60640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb158142520 con 0x7fb15813f710 2024-03-25T18:26:55.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.134+0000 7fb15ea60640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1580b28a0 con 0x7fb1581430c0 2024-03-25T18:26:55.137 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.134+0000 7fb15ea60640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1580b2a30 con 0x7fb1580a98f0 2024-03-25T18:26:55.137 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.134+0000 7fb15ea60640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:55.137 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.134+0000 7fb15ea60640 10 monclient(hunting): authenticate will time out at 7433.302538s 2024-03-25T18:26:55.141 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.140+0000 7fb15cffd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb1581430c0 0x7fb15814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:54904/0 (socket says 172.21.15.99:54904) 2024-03-25T18:26:55.141 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.140+0000 7fb15cffd640 1 -- 172.21.15.99:0/3181177475 learned_addr learned my addr 172.21.15.99:0/3181177475 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:55.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.150+0000 7fb156ffd640 1 -- 172.21.15.99:0/3181177475 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260995075 0 0) 0x7fb1580b28a0 con 0x7fb1581430c0 2024-03-25T18:26:55.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.150+0000 7fb156ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.150+0000 7fb156ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.150+0000 7fb156ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.150+0000 7fb156ffd640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:26:55.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.150+0000 7fb156ffd640 1 -- 172.21.15.99:0/3181177475 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb140003520 con 0x7fb1581430c0 2024-03-25T18:26:55.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.150+0000 7fb156ffd640 1 -- 172.21.15.99:0/3181177475 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524293524 0 0) 0x7fb158142520 con 0x7fb15813f710 2024-03-25T18:26:55.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.150+0000 7fb156ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.150+0000 7fb156ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.150+0000 7fb156ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.150+0000 7fb156ffd640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:26:55.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.150+0000 7fb156ffd640 1 -- 172.21.15.99:0/3181177475 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1580b28a0 con 0x7fb15813f710 2024-03-25T18:26:55.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 1 -- 172.21.15.99:0/3181177475 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058493272 0 0) 0x7fb1580b28a0 con 0x7fb15813f710 2024-03-25T18:26:55.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 1 -- 172.21.15.99:0/3181177475 >> v1:172.21.15.99:6790/0 conn(0x7fb1580a98f0 legacy=0x7fb1580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 1 -- 172.21.15.99:0/3181177475 >> v1:172.21.15.190:6789/0 conn(0x7fb1581430c0 legacy=0x7fb15814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:55.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 1 monclient: found mon.noname-a 2024-03-25T18:26:55.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:55.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:26:55.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 1 -- 172.21.15.99:0/3181177475 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1580b44c0 con 0x7fb15813f710 2024-03-25T18:26:55.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:55.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:55.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:55.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 1 -- 172.21.15.99:0/3181177475 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb14c001670 con 0x7fb15813f710 2024-03-25T18:26:55.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb15ea60640 5 monclient: authenticate success, global_id 4287 2024-03-25T18:26:55.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb15ea60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:55.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:55.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:55.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb156ffd640 10 monclient: dump: 2024-03-25T18:26:55.160 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:55.160 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:55.160 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:55.160 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:55.160 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:55.161 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:55.161 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:55.161 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:55.161 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:55.161 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:55.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.158+0000 7fb15ea60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:55.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.187+0000 7fb156ffd640 1 -- 172.21.15.99:0/3181177475 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb14c0017d0 con 0x7fb15813f710 2024-03-25T18:26:55.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.187+0000 7fb156ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:55.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.187+0000 7fb156ffd640 1 -- 172.21.15.99:0/3181177475 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb14c001ba0 con 0x7fb15813f710 2024-03-25T18:26:55.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.187+0000 7fb156ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:55.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.187+0000 7fb156ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:55.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.187+0000 7fb156ffd640 10 monclient: dump: 2024-03-25T18:26:55.189 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:55.189 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:55.189 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:55.189 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:55.189 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:55.189 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:55.189 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:55.189 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:55.189 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:55.189 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:55.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.187+0000 7fb15ea60640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:55.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.187+0000 7fb15ea60640 4 set_mon_vals no callback set 2024-03-25T18:26:55.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.188+0000 7fb15ea60640 10 monclient: shutdown 2024-03-25T18:26:55.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.188+0000 7fb15ea60640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:55.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.188+0000 7fb15ea60640 1 -- 172.21.15.99:0/3181177475 >> v1:172.21.15.99:6789/0 conn(0x7fb15813f710 legacy=0x7fb1581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.188+0000 7fb15ea60640 1 -- 172.21.15.99:0/3181177475 shutdown_connections 2024-03-25T18:26:55.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.188+0000 7fb15ea60640 1 -- 172.21.15.99:0/3181177475 >> 172.21.15.99:0/3181177475 conn(0x7fb1580b7ea0 msgr2=0x7fb1580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:55.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.188+0000 7fb15ea60640 1 -- 172.21.15.99:0/3181177475 shutdown_connections 2024-03-25T18:26:55.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.188+0000 7fb15ea60640 1 -- 172.21.15.99:0/3181177475 wait complete. 2024-03-25T18:26:55.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.189+0000 7fb15ea60640 10 monclient: build_initial_monmap 2024-03-25T18:26:55.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.189+0000 7fb15ea60640 10 monclient: monmap: 2024-03-25T18:26:55.192 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:26:55.192 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:55.192 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:26:55.192 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:26:55.192 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:55.192 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.189+0000 7fb15ea60640 1 librados: starting msgr at 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.189+0000 7fb15ea60640 1 librados: starting objecter 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.189+0000 7fb15ea60640 1 Processor -- start 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.189+0000 7fb15ea60640 1 -- start start 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.189+0000 7fb15ea60640 1 librados: setting wanted keys 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.189+0000 7fb15ea60640 1 librados: calling monclient init 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.189+0000 7fb15ea60640 10 monclient: init 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.190+0000 7fb15ea60640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.190+0000 7fb15ea60640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.190+0000 7fb15ea60640 10 monclient(hunting): picked mon.noname-b con 0x7fb1580a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.190+0000 7fb15ea60640 10 monclient(hunting): picked mon.noname-c con 0x7fb15813f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:26:55.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.190+0000 7fb15ea60640 10 monclient(hunting): picked mon.noname-a con 0x7fb1581430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:26:55.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.190+0000 7fb15ea60640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1581d7830 con 0x7fb1581430c0 2024-03-25T18:26:55.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.190+0000 7fb15ea60640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1581d7500 con 0x7fb1580a98f0 2024-03-25T18:26:55.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.190+0000 7fb15ea60640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb158137970 con 0x7fb15813f710 2024-03-25T18:26:55.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.190+0000 7fb15ea60640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:55.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.190+0000 7fb15ea60640 10 monclient(hunting): authenticate will time out at 7433.358638s 2024-03-25T18:26:55.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.200+0000 7fb157fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb1580a98f0 0x7fb158138710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:54932/0 (socket says 172.21.15.99:54932) 2024-03-25T18:26:55.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.200+0000 7fb157fff640 1 -- 172.21.15.99:0/3629550798 learned_addr learned my addr 172.21.15.99:0/3629550798 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:26:55.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.202+0000 7fb154ff9640 10 client.?.objecter ms_handle_connect 0x7fb1580a98f0 2024-03-25T18:26:55.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.202+0000 7fb154ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:55.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.204+0000 7fb154ff9640 10 client.?.objecter ms_handle_connect 0x7fb15813f710 2024-03-25T18:26:55.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.204+0000 7fb154ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:55.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.204+0000 7fb154ff9640 10 client.?.objecter ms_handle_connect 0x7fb1581430c0 2024-03-25T18:26:55.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.204+0000 7fb154ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:55.210 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.209+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122883746 0 0) 0x7fb1581d7830 con 0x7fb1581430c0 2024-03-25T18:26:55.210 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.209+0000 7fb154ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.210 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.209+0000 7fb154ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.211 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.209+0000 7fb154ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.211 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.209+0000 7fb154ff9640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:26:55.211 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.209+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb144004eb0 con 0x7fb1581430c0 2024-03-25T18:26:55.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.215+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604308606 0 0) 0x7fb158137970 con 0x7fb15813f710 2024-03-25T18:26:55.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.215+0000 7fb154ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.215+0000 7fb154ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.215+0000 7fb154ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.215+0000 7fb154ff9640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:26:55.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.215+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1581d7830 con 0x7fb15813f710 2024-03-25T18:26:55.220 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.219+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1368639701 0 0) 0x7fb144004eb0 con 0x7fb1581430c0 2024-03-25T18:26:55.220 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.219+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb158137970 con 0x7fb1581430c0 2024-03-25T18:26:55.220 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.219+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb150001b90 con 0x7fb1581430c0 2024-03-25T18:26:55.221 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.219+0000 7fb154ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:55.221 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.219+0000 7fb154ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:55.221 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.219+0000 7fb154ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:55.221 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:55.221 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:55.221 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:55.221 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:55.221 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:55.221 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:55.221 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:55.221 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:55.221 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:55.221 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:55.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.219+0000 7fb154ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:55.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.220+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035610141 0 0) 0x7fb1581d7500 con 0x7fb1580a98f0 2024-03-25T18:26:55.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.220+0000 7fb154ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.220+0000 7fb154ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.220+0000 7fb154ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.220+0000 7fb154ff9640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:26:55.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.220+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb144004eb0 con 0x7fb1580a98f0 2024-03-25T18:26:55.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.232+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 395087412 0 0) 0x7fb1581d7830 con 0x7fb15813f710 2024-03-25T18:26:55.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.232+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1581d7500 con 0x7fb15813f710 2024-03-25T18:26:55.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.232+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb14c001ae0 con 0x7fb15813f710 2024-03-25T18:26:55.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.232+0000 7fb154ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:55.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.232+0000 7fb154ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.232+0000 7fb154ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.232+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2571195500 0 0) 0x7fb144004eb0 con 0x7fb1580a98f0 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.232+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1581d7830 con 0x7fb1580a98f0 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.232+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb1480018f0 con 0x7fb1580a98f0 2024-03-25T18:26:55.235 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.232+0000 7fb154ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:55.236 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.232+0000 7fb154ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:55.236 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.232+0000 7fb154ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:55.236 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:55.236 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:55.236 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:55.236 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:55.236 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:55.236 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:55.236 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:55.236 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:55.237 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:55.237 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:55.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4070026009 0 0) 0x7fb1581d7500 con 0x7fb15813f710 2024-03-25T18:26:55.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 >> v1:172.21.15.190:6789/0 conn(0x7fb1580a98f0 legacy=0x7fb158138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 >> v1:172.21.15.99:6789/0 conn(0x7fb1581430c0 legacy=0x7fb1581e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb154ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:55.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb154ff9640 1 monclient: found mon.c 2024-03-25T18:26:55.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb154ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:55.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb154ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:26:55.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb158137b00 con 0x7fb15813f710 2024-03-25T18:26:55.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb154ff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:55.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb154ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:55.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb154ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:55.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb15ea60640 5 monclient: authenticate success, global_id 4301 2024-03-25T18:26:55.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb15ea60640 10 monclient: _renew_subs 2024-03-25T18:26:55.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb15ea60640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:26:55.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb15ea60640 1 -- 172.21.15.99:0/3629550798 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb158135080 con 0x7fb15813f710 2024-03-25T18:26:55.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb15ea60640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:55.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb15ea60640 10 monclient: _renew_subs 2024-03-25T18:26:55.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb15ea60640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:26:55.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb15ea60640 1 -- 172.21.15.99:0/3629550798 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1581e36d0 con 0x7fb15813f710 2024-03-25T18:26:55.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.246+0000 7fb15ea60640 1 librados: init done 2024-03-25T18:26:55.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.247+0000 7fb15ea60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:55.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.247+0000 7fb15ea60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:55.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.247+0000 7fb15ea60640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:26:55.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.247+0000 7fb15ea60640 1 -- 172.21.15.99:0/3629550798 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1581d7690 con 0x7fb15813f710 2024-03-25T18:26:55.551 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.550+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb14c0038a0 con 0x7fb15813f710 2024-03-25T18:26:55.552 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.550+0000 7fb154ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:55.552 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.550+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb14c003a10 con 0x7fb15813f710 2024-03-25T18:26:55.552 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.550+0000 7fb154ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:55.552 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.550+0000 7fb154ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:55.552 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.550+0000 7fb154ff9640 10 monclient: dump: 2024-03-25T18:26:55.552 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:26:55.552 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:26:55.552 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:55.552 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:26:55.552 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:55.552 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:26:55.553 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:26:55.553 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:26:55.553 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:26:55.553 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:26:55.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.550+0000 7fb1567fc640 4 set_mon_vals no callback set 2024-03-25T18:26:55.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.551+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fb14c0199e0 con 0x7fb15813f710 2024-03-25T18:26:55.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.551+0000 7fb157fff640 10 monclient: get_auth_request con 0x7fb14406e0c0 auth_method 0 2024-03-25T18:26:55.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.552+0000 7fb154ff9640 10 client.4301.objecter ms_handle_connect 0x7fb14406e0c0 2024-03-25T18:26:55.558 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.557+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fb14c053480 con 0x7fb15813f710 2024-03-25T18:26:55.558 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.557+0000 7fb154ff9640 10 client.4301.objecter ms_dispatch 0x7fb158000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:55.558 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.557+0000 7fb154ff9640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:55.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.557+0000 7fb154ff9640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:55.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.557+0000 7fb154ff9640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-25T18:26:55.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.557+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb14c04ef90 con 0x7fb15813f710 2024-03-25T18:26:55.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.557+0000 7fb154ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:55.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.557+0000 7fb154ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:55.804 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.802+0000 7fb15ea60640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:55.804 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.802+0000 7fb15ea60640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:55.804 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.802+0000 7fb15ea60640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:26:55.804 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.802+0000 7fb15ea60640 1 -- 172.21.15.99:0/3629550798 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1580627b0 con 0x7fb15813f710 2024-03-25T18:26:55.845 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.843+0000 7fb154ff9640 1 -- 172.21.15.99:0/3629550798 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fb14c050ec0 con 0x7fb15813f710 2024-03-25T18:26:55.845 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.844+0000 7fb154ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:55.845 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.844+0000 7fb154ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:55.845 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:26:55.845 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.846+0000 7fb15ea60640 10 librados: watch_flush enter 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.846+0000 7fb15ea60640 10 librados: watch_flush exit 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.846+0000 7fb15ea60640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.846+0000 7fb15ea60640 10 client.4301.objecter successfully canceled tick 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.846+0000 7fb15ea60640 1 -- 172.21.15.99:0/3629550798 >> v1:172.21.15.190:6800/4220409699 conn(0x7fb14406e0c0 legacy=0x7fb144070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.847+0000 7fb15ea60640 10 monclient: shutdown 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.847+0000 7fb15ea60640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.847+0000 7fb15ea60640 1 -- 172.21.15.99:0/3629550798 >> v1:172.21.15.99:6790/0 conn(0x7fb15813f710 legacy=0x7fb1581d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.847+0000 7fb15ea60640 1 -- 172.21.15.99:0/3629550798 shutdown_connections 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.847+0000 7fb15ea60640 1 -- 172.21.15.99:0/3629550798 >> 172.21.15.99:0/3629550798 conn(0x7fb1580b7ea0 msgr2=0x7fb158074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:55.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.847+0000 7fb15ea60640 1 -- 172.21.15.99:0/3629550798 shutdown_connections 2024-03-25T18:26:55.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.847+0000 7fb15ea60640 1 -- 172.21.15.99:0/3629550798 wait complete. 2024-03-25T18:26:55.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:26:55.847+0000 7fb15ea60640 1 librados: shutdown 2024-03-25T18:26:55.860 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:02.669 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:02.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.781+0000 7f5d63fd0640 10 monclient: get_monmap_and_config 2024-03-25T18:27:02.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.781+0000 7f5d63fd0640 10 monclient: build_initial_monmap 2024-03-25T18:27:02.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.781+0000 7f5d63fd0640 10 monclient: monmap: 2024-03-25T18:27:02.782 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:27:02.782 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:02.782 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:27:02.782221+0000 2024-03-25T18:27:02.782 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:27:02.782221+0000 2024-03-25T18:27:02.782 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:02.782 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:02.782 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:27:02.783 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:27:02.783 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:27:02.783 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:02.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.781+0000 7f5d63fd0640 1 Processor -- start 2024-03-25T18:27:02.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.782+0000 7f5d63fd0640 1 -- start start 2024-03-25T18:27:02.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.782+0000 7f5d63fd0640 10 monclient: init 2024-03-25T18:27:02.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.782+0000 7f5d63fd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:02.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.782+0000 7f5d63fd0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:02.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.782+0000 7f5d63fd0640 10 monclient(hunting): picked mon.noname-a con 0x7f5d5c0a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:02.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.782+0000 7f5d63fd0640 10 monclient(hunting): picked mon.noname-c con 0x7f5d5c13f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:02.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.782+0000 7f5d63fd0640 10 monclient(hunting): picked mon.noname-b con 0x7f5d5c1430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:02.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.782+0000 7f5d63fd0640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5c142520 con 0x7f5d5c0a98f0 2024-03-25T18:27:02.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.782+0000 7f5d63fd0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5c0b28a0 con 0x7f5d5c13f710 2024-03-25T18:27:02.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.782+0000 7f5d63fd0640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5c0b2a30 con 0x7f5d5c1430c0 2024-03-25T18:27:02.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.782+0000 7f5d63fd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:02.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.782+0000 7f5d63fd0640 10 monclient(hunting): authenticate will time out at 7440.950585s 2024-03-25T18:27:02.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.793+0000 7f5d6156b640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f5d5c13f710 0x7f5d5c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:59700/0 (socket says 172.21.15.99:59700) 2024-03-25T18:27:02.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.793+0000 7f5d6156b640 1 -- 172.21.15.99:0/2522254499 learned_addr learned my addr 172.21.15.99:0/2522254499 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:27:02.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.808+0000 7f5d60d6a640 1 -- 172.21.15.99:0/2522254499 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609248323 0 0) 0x7f5d5c0b28a0 con 0x7f5d5c13f710 2024-03-25T18:27:02.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.808+0000 7f5d60d6a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:02.810 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.808+0000 7f5d60d6a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:02.810 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.808+0000 7f5d60d6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:02.810 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.808+0000 7f5d60d6a640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:27:02.810 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.808+0000 7f5d60d6a640 1 -- 172.21.15.99:0/2522254499 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d48003520 con 0x7f5d5c13f710 2024-03-25T18:27:02.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 1 -- 172.21.15.99:0/2522254499 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2435117881 0 0) 0x7f5d48003520 con 0x7f5d5c13f710 2024-03-25T18:27:02.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 1 -- 172.21.15.99:0/2522254499 >> v1:172.21.15.99:6790/0 conn(0x7f5d5c1430c0 legacy=0x7f5d5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:02.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 1 -- 172.21.15.99:0/2522254499 >> v1:172.21.15.99:6789/0 conn(0x7f5d5c0a98f0 legacy=0x7f5d5c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:02.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 10 monclient: _finish_hunting 0 2024-03-25T18:27:02.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 1 monclient: found mon.noname-c 2024-03-25T18:27:02.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:02.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:27:02.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 1 -- 172.21.15.99:0/2522254499 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c0b44c0 con 0x7f5d5c13f710 2024-03-25T18:27:02.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 10 monclient: _finish_auth 0 2024-03-25T18:27:02.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 10 monclient: _check_auth_tickets 2024-03-25T18:27:02.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:02.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 1 -- 172.21.15.99:0/2522254499 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5d4c001690 con 0x7f5d5c13f710 2024-03-25T18:27:02.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d63fd0640 5 monclient: authenticate success, global_id 4303 2024-03-25T18:27:02.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d63fd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:02.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:02.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:02.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d60d6a640 10 monclient: dump: 2024-03-25T18:27:02.819 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:02.819 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:02.819 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:02.819 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:02.819 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:02.819 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:02.819 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:02.819 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:02.819 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:02.819 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:02.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.815+0000 7f5d63fd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:02.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.827+0000 7f5d60d6a640 1 -- 172.21.15.99:0/2522254499 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d4c0017f0 con 0x7f5d5c13f710 2024-03-25T18:27:02.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.827+0000 7f5d60d6a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:02.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.827+0000 7f5d60d6a640 1 -- 172.21.15.99:0/2522254499 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5d4c001c00 con 0x7f5d5c13f710 2024-03-25T18:27:02.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.827+0000 7f5d60d6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:02.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.827+0000 7f5d60d6a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:02.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.827+0000 7f5d60d6a640 10 monclient: dump: 2024-03-25T18:27:02.829 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:02.829 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:02.829 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:02.830 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:02.830 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:02.830 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:02.830 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:02.830 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:02.830 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:02.830 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:02.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.827+0000 7f5d63fd0640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:02.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.827+0000 7f5d63fd0640 4 set_mon_vals no callback set 2024-03-25T18:27:02.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.828+0000 7f5d63fd0640 10 monclient: shutdown 2024-03-25T18:27:02.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.828+0000 7f5d63fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:02.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.828+0000 7f5d63fd0640 1 -- 172.21.15.99:0/2522254499 >> v1:172.21.15.190:6789/0 conn(0x7f5d5c13f710 legacy=0x7f5d5c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:02.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.828+0000 7f5d63fd0640 1 -- 172.21.15.99:0/2522254499 shutdown_connections 2024-03-25T18:27:02.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.828+0000 7f5d63fd0640 1 -- 172.21.15.99:0/2522254499 >> 172.21.15.99:0/2522254499 conn(0x7f5d5c0b7ea0 msgr2=0x7f5d5c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:02.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.828+0000 7f5d63fd0640 1 -- 172.21.15.99:0/2522254499 shutdown_connections 2024-03-25T18:27:02.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.828+0000 7f5d63fd0640 1 -- 172.21.15.99:0/2522254499 wait complete. 2024-03-25T18:27:02.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.829+0000 7f5d63fd0640 10 monclient: build_initial_monmap 2024-03-25T18:27:02.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.829+0000 7f5d63fd0640 10 monclient: monmap: 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.829+0000 7f5d63fd0640 1 librados: starting msgr at 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.829+0000 7f5d63fd0640 1 librados: starting objecter 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.829+0000 7f5d63fd0640 1 Processor -- start 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.830+0000 7f5d63fd0640 1 -- start start 2024-03-25T18:27:02.833 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.830+0000 7f5d63fd0640 1 librados: setting wanted keys 2024-03-25T18:27:02.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.830+0000 7f5d63fd0640 1 librados: calling monclient init 2024-03-25T18:27:02.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.830+0000 7f5d63fd0640 10 monclient: init 2024-03-25T18:27:02.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.830+0000 7f5d63fd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:02.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.830+0000 7f5d63fd0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:02.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.830+0000 7f5d63fd0640 10 monclient(hunting): picked mon.noname-c con 0x7f5d5c0a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:02.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.830+0000 7f5d63fd0640 10 monclient(hunting): picked mon.noname-b con 0x7f5d5c13f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:02.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.830+0000 7f5d63fd0640 10 monclient(hunting): picked mon.noname-a con 0x7f5d5c1430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:02.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.830+0000 7f5d63fd0640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5c1d8370 con 0x7f5d5c1430c0 2024-03-25T18:27:02.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.830+0000 7f5d63fd0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5c1d7df0 con 0x7f5d5c13f710 2024-03-25T18:27:02.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.830+0000 7f5d63fd0640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5c1d7f80 con 0x7f5d5c0a98f0 2024-03-25T18:27:02.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.830+0000 7f5d63fd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:02.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.830+0000 7f5d63fd0640 10 monclient(hunting): authenticate will time out at 7440.998734s 2024-03-25T18:27:02.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.840+0000 7f5d61d6c640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f5d5c0a98f0 0x7f5d5c1d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:48118/0 (socket says 172.21.15.99:48118) 2024-03-25T18:27:02.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.840+0000 7f5d61d6c640 1 -- 172.21.15.99:0/1617644191 learned_addr learned my addr 172.21.15.99:0/1617644191 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:27:02.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.841+0000 7f5d527fc640 10 client.?.objecter ms_handle_connect 0x7f5d5c0a98f0 2024-03-25T18:27:02.842 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.841+0000 7f5d527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:02.843 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.843+0000 7f5d527fc640 10 client.?.objecter ms_handle_connect 0x7f5d5c1430c0 2024-03-25T18:27:02.844 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.843+0000 7f5d527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:02.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.850+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030493953 0 0) 0x7f5d5c1d7f80 con 0x7f5d5c0a98f0 2024-03-25T18:27:02.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.850+0000 7f5d527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:02.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.850+0000 7f5d527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:02.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.850+0000 7f5d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:02.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.850+0000 7f5d527fc640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:27:02.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.850+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d40004eb0 con 0x7f5d5c0a98f0 2024-03-25T18:27:02.859 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.855+0000 7f5d527fc640 10 client.?.objecter ms_handle_connect 0x7f5d5c13f710 2024-03-25T18:27:02.859 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.855+0000 7f5d527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:02.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.859+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032425375 0 0) 0x7f5d5c1d8370 con 0x7f5d5c1430c0 2024-03-25T18:27:02.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.859+0000 7f5d527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:02.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.859+0000 7f5d527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:02.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.859+0000 7f5d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:02.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.859+0000 7f5d527fc640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:27:02.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.859+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d5c1d7f80 con 0x7f5d5c1430c0 2024-03-25T18:27:02.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.861+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920309401 0 0) 0x7f5d5c1d7df0 con 0x7f5d5c13f710 2024-03-25T18:27:02.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.861+0000 7f5d527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:02.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.861+0000 7f5d527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:02.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.861+0000 7f5d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:02.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.861+0000 7f5d527fc640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:27:02.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.861+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d5c1d8370 con 0x7f5d5c13f710 2024-03-25T18:27:02.875 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.874+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1111867030 0 0) 0x7f5d40004eb0 con 0x7f5d5c0a98f0 2024-03-25T18:27:02.875 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.874+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d5c1d7df0 con 0x7f5d5c0a98f0 2024-03-25T18:27:02.875 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.874+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5d58001a30 con 0x7f5d5c0a98f0 2024-03-25T18:27:02.875 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.874+0000 7f5d527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:02.875 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.874+0000 7f5d527fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:02.875 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.874+0000 7f5d527fc640 10 monclient(hunting): dump: 2024-03-25T18:27:02.875 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:02.875 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:02.875 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:02.875 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:02.875 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:02.875 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:02.876 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:02.876 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:02.876 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:02.876 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:02.876 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.874+0000 7f5d527fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:02.876 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.874+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2057686886 0 0) 0x7f5d5c1d7f80 con 0x7f5d5c1430c0 2024-03-25T18:27:02.877 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.874+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d40004eb0 con 0x7f5d5c1430c0 2024-03-25T18:27:02.877 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.874+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5d54001b00 con 0x7f5d5c1430c0 2024-03-25T18:27:02.877 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.874+0000 7f5d527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:02.877 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.874+0000 7f5d527fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:02.877 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.874+0000 7f5d527fc640 10 monclient(hunting): dump: 2024-03-25T18:27:02.877 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:02.877 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:02.877 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:02.877 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:02.877 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:02.877 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:02.877 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:02.877 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:02.877 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:02.878 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:02.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.884+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1314690953 0 0) 0x7f5d5c1d8370 con 0x7f5d5c13f710 2024-03-25T18:27:02.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.884+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d5c1d7f80 con 0x7f5d5c13f710 2024-03-25T18:27:02.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.884+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5d4c001b00 con 0x7f5d5c13f710 2024-03-25T18:27:02.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.884+0000 7f5d527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:02.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.884+0000 7f5d527fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:02.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.884+0000 7f5d527fc640 10 monclient(hunting): dump: 2024-03-25T18:27:02.886 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:02.886 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:02.886 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:02.886 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:02.886 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:02.886 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:02.886 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:02.886 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:02.886 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:02.886 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:02.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1042806761 0 0) 0x7f5d5c1d7df0 con 0x7f5d5c0a98f0 2024-03-25T18:27:02.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 >> v1:172.21.15.190:6789/0 conn(0x7f5d5c13f710 legacy=0x7f5d5c132f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:02.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 >> v1:172.21.15.99:6789/0 conn(0x7f5d5c1430c0 legacy=0x7f5d5c1d4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:02.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d527fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:02.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d527fc640 1 monclient: found mon.c 2024-03-25T18:27:02.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:02.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d527fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:02.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c1d8110 con 0x7f5d5c0a98f0 2024-03-25T18:27:02.898 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d527fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:02.898 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d527fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:02.898 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:02.898 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d63fd0640 5 monclient: authenticate success, global_id 4307 2024-03-25T18:27:02.898 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d63fd0640 10 monclient: _renew_subs 2024-03-25T18:27:02.898 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d63fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:02.898 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d63fd0640 1 -- 172.21.15.99:0/1617644191 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5c13b910 con 0x7f5d5c0a98f0 2024-03-25T18:27:02.898 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d63fd0640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:02.898 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d63fd0640 10 monclient: _renew_subs 2024-03-25T18:27:02.898 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d63fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:02.898 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d63fd0640 1 -- 172.21.15.99:0/1617644191 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5c1336a0 con 0x7f5d5c0a98f0 2024-03-25T18:27:02.898 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.896+0000 7f5d63fd0640 1 librados: init done 2024-03-25T18:27:02.899 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.897+0000 7f5d63fd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:02.899 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.897+0000 7f5d63fd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:02.899 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.897+0000 7f5d63fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:02.899 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:02.897+0000 7f5d63fd0640 1 -- 172.21.15.99:0/1617644191 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5c0627b0 con 0x7f5d5c0a98f0 2024-03-25T18:27:03.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.163+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d58001c70 con 0x7f5d5c0a98f0 2024-03-25T18:27:03.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.163+0000 7f5d527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:03.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.163+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5d58003a60 con 0x7f5d5c0a98f0 2024-03-25T18:27:03.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.163+0000 7f5d527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:03.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.163+0000 7f5d527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:03.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.163+0000 7f5d527fc640 10 monclient: dump: 2024-03-25T18:27:03.164 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:03.165 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:03.165 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:03.165 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:03.165 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:03.165 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:03.165 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:03.165 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:03.165 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:03.165 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:03.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.163+0000 7f5d53fff640 4 set_mon_vals no callback set 2024-03-25T18:27:03.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.258+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f5d58003bc0 con 0x7f5d5c0a98f0 2024-03-25T18:27:03.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.258+0000 7f5d6156b640 10 monclient: get_auth_request con 0x7f5d4006e110 auth_method 0 2024-03-25T18:27:03.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.259+0000 7f5d527fc640 10 client.4307.objecter ms_handle_connect 0x7f5d4006e110 2024-03-25T18:27:03.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.273+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f5d580548d0 con 0x7f5d5c0a98f0 2024-03-25T18:27:03.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.273+0000 7f5d527fc640 10 client.4307.objecter ms_dispatch 0x7f5d5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:03.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.273+0000 7f5d527fc640 3 client.4307.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:03.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.273+0000 7f5d527fc640 3 client.4307.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:03.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.274+0000 7f5d527fc640 20 client.4307.objecter dump_active .. 0 homeless 2024-03-25T18:27:03.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.274+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d58050210 con 0x7f5d5c0a98f0 2024-03-25T18:27:03.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.274+0000 7f5d527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:03.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.274+0000 7f5d527fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:03.527 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.526+0000 7f5d63fd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:03.527 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.526+0000 7f5d63fd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:03.527 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.526+0000 7f5d63fd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:03.527 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.526+0000 7f5d63fd0640 1 -- 172.21.15.99:0/1617644191 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d5c0008d0 con 0x7f5d5c0a98f0 2024-03-25T18:27:03.562 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.561+0000 7f5d527fc640 1 -- 172.21.15.99:0/1617644191 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f5d58052310 con 0x7f5d5c0a98f0 2024-03-25T18:27:03.562 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.561+0000 7f5d527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:03.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.561+0000 7f5d527fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:03.563 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:27:03.563 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:03.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.564+0000 7f5d63fd0640 10 librados: watch_flush enter 2024-03-25T18:27:03.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.564+0000 7f5d63fd0640 10 librados: watch_flush exit 2024-03-25T18:27:03.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.564+0000 7f5d63fd0640 20 client.4307.objecter shutdown clearing up homeless session... 2024-03-25T18:27:03.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.564+0000 7f5d63fd0640 10 client.4307.objecter successfully canceled tick 2024-03-25T18:27:03.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.564+0000 7f5d63fd0640 1 -- 172.21.15.99:0/1617644191 >> v1:172.21.15.190:6800/4220409699 conn(0x7f5d4006e110 legacy=0x7f5d400705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.564+0000 7f5d63fd0640 10 monclient: shutdown 2024-03-25T18:27:03.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.564+0000 7f5d63fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:03.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.564+0000 7f5d63fd0640 1 -- 172.21.15.99:0/1617644191 >> v1:172.21.15.99:6790/0 conn(0x7f5d5c0a98f0 legacy=0x7f5d5c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.564+0000 7f5d63fd0640 1 -- 172.21.15.99:0/1617644191 shutdown_connections 2024-03-25T18:27:03.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.564+0000 7f5d63fd0640 1 -- 172.21.15.99:0/1617644191 >> 172.21.15.99:0/1617644191 conn(0x7f5d5c0b7ea0 msgr2=0x7f5d5c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:03.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.564+0000 7f5d63fd0640 1 -- 172.21.15.99:0/1617644191 shutdown_connections 2024-03-25T18:27:03.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.564+0000 7f5d63fd0640 1 -- 172.21.15.99:0/1617644191 wait complete. 2024-03-25T18:27:03.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:03.564+0000 7f5d63fd0640 1 librados: shutdown 2024-03-25T18:27:03.576 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:10.383 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:10.499 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.498+0000 7f1ef4b7c640 10 monclient: get_monmap_and_config 2024-03-25T18:27:10.500 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.498+0000 7f1ef4b7c640 10 monclient: build_initial_monmap 2024-03-25T18:27:10.500 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.498+0000 7f1ef4b7c640 10 monclient: monmap: 2024-03-25T18:27:10.500 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:27:10.500 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:10.500 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:27:10.499925+0000 2024-03-25T18:27:10.500 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:27:10.499925+0000 2024-03-25T18:27:10.500 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:10.500 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:10.500 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:27:10.500 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:27:10.500 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:27:10.501 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:10.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.499+0000 7f1ef4b7c640 1 Processor -- start 2024-03-25T18:27:10.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.499+0000 7f1ef4b7c640 1 -- start start 2024-03-25T18:27:10.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.499+0000 7f1ef4b7c640 10 monclient: init 2024-03-25T18:27:10.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.500+0000 7f1ef4b7c640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:10.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.500+0000 7f1ef4b7c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:10.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.500+0000 7f1ef4b7c640 10 monclient(hunting): picked mon.noname-b con 0x7f1ef00b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:10.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.500+0000 7f1ef4b7c640 10 monclient(hunting): picked mon.noname-a con 0x7f1ef00b1730 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:10.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.500+0000 7f1ef4b7c640 10 monclient(hunting): picked mon.noname-c con 0x7f1ef014be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:10.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.500+0000 7f1ef4b7c640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef00af940 con 0x7f1ef00b1730 2024-03-25T18:27:10.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.500+0000 7f1ef4b7c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef00b0a70 con 0x7f1ef014be90 2024-03-25T18:27:10.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.500+0000 7f1ef4b7c640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef00b0d50 con 0x7f1ef00b4d60 2024-03-25T18:27:10.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.500+0000 7f1ef4b7c640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:10.503 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.500+0000 7f1ef4b7c640 10 monclient(hunting): authenticate will time out at 7448.668389s 2024-03-25T18:27:10.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.509+0000 7f1ee7fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f1ef00b1730 0x7f1ef00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:36846/0 (socket says 172.21.15.99:36846) 2024-03-25T18:27:10.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.509+0000 7f1ee7fff640 1 -- 172.21.15.99:0/486068658 learned_addr learned my addr 172.21.15.99:0/486068658 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:27:10.520 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.519+0000 7f1eedd9b640 1 -- 172.21.15.99:0/486068658 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973904927 0 0) 0x7f1ef00af940 con 0x7f1ef00b1730 2024-03-25T18:27:10.520 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.519+0000 7f1eedd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:27:10.520 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.519+0000 7f1eedd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:10.520 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.519+0000 7f1eedd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:10.520 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.519+0000 7f1eedd9b640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:27:10.520 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.519+0000 7f1eedd9b640 1 -- 172.21.15.99:0/486068658 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1edc003520 con 0x7f1ef00b1730 2024-03-25T18:27:10.531 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.530+0000 7f1eedd9b640 1 -- 172.21.15.99:0/486068658 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555392235 0 0) 0x7f1ef00b0a70 con 0x7f1ef014be90 2024-03-25T18:27:10.531 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.530+0000 7f1eedd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:27:10.531 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.530+0000 7f1eedd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:10.531 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.530+0000 7f1eedd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:10.531 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.530+0000 7f1eedd9b640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:27:10.531 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.530+0000 7f1eedd9b640 1 -- 172.21.15.99:0/486068658 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef00af940 con 0x7f1ef014be90 2024-03-25T18:27:10.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.534+0000 7f1eedd9b640 1 -- 172.21.15.99:0/486068658 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1693016745 0 0) 0x7f1ef00af940 con 0x7f1ef014be90 2024-03-25T18:27:10.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.534+0000 7f1eedd9b640 1 -- 172.21.15.99:0/486068658 >> v1:172.21.15.99:6790/0 conn(0x7f1ef00b4d60 legacy=0x7f1ef013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:10.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.534+0000 7f1eedd9b640 1 -- 172.21.15.99:0/486068658 >> v1:172.21.15.99:6789/0 conn(0x7f1ef00b1730 legacy=0x7f1ef00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:10.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.534+0000 7f1eedd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:27:10.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.534+0000 7f1eedd9b640 1 monclient: found mon.noname-c 2024-03-25T18:27:10.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.534+0000 7f1eedd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:10.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.534+0000 7f1eedd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:27:10.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.534+0000 7f1eedd9b640 1 -- 172.21.15.99:0/486068658 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef00b1030 con 0x7f1ef014be90 2024-03-25T18:27:10.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.534+0000 7f1eedd9b640 10 monclient: _finish_auth 0 2024-03-25T18:27:10.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.534+0000 7f1eedd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:27:10.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.534+0000 7f1eedd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:10.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.535+0000 7f1eedd9b640 1 -- 172.21.15.99:0/486068658 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f1ee80019f0 con 0x7f1ef014be90 2024-03-25T18:27:10.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.535+0000 7f1eedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:10.537 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.535+0000 7f1eedd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:10.537 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.535+0000 7f1eedd9b640 10 monclient: dump: 2024-03-25T18:27:10.537 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:10.537 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:10.537 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:10.537 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:10.537 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:10.537 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:10.537 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:10.537 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:10.537 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:10.537 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:10.537 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.535+0000 7f1ef4b7c640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:27:10.537 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.535+0000 7f1ef4b7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:10.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.554+0000 7f1eedd9b640 1 -- 172.21.15.99:0/486068658 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ee8001b50 con 0x7f1ef014be90 2024-03-25T18:27:10.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.554+0000 7f1eedd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:10.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.554+0000 7f1eedd9b640 1 -- 172.21.15.99:0/486068658 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f1ee8001f20 con 0x7f1ef014be90 2024-03-25T18:27:10.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.554+0000 7f1eedd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:10.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.554+0000 7f1eedd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:10.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.554+0000 7f1eedd9b640 10 monclient: dump: 2024-03-25T18:27:10.555 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:10.555 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:10.556 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:10.556 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:10.556 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:10.556 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:10.556 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:10.556 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:10.556 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:10.556 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:10.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.554+0000 7f1ef4b7c640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:10.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.554+0000 7f1ef4b7c640 4 set_mon_vals no callback set 2024-03-25T18:27:10.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.555+0000 7f1ef4b7c640 10 monclient: shutdown 2024-03-25T18:27:10.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.555+0000 7f1ef4b7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:10.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.555+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/486068658 >> v1:172.21.15.190:6789/0 conn(0x7f1ef014be90 legacy=0x7f1ef014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:10.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.555+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/486068658 shutdown_connections 2024-03-25T18:27:10.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.555+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/486068658 >> 172.21.15.99:0/486068658 conn(0x7f1ef00617a0 msgr2=0x7f1ef0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:10.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.555+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/486068658 shutdown_connections 2024-03-25T18:27:10.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.555+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/486068658 wait complete. 2024-03-25T18:27:10.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.556+0000 7f1ef4b7c640 10 monclient: build_initial_monmap 2024-03-25T18:27:10.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.556+0000 7f1ef4b7c640 10 monclient: monmap: 2024-03-25T18:27:10.559 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:27:10.559 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.556+0000 7f1ef4b7c640 1 librados: starting msgr at 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.556+0000 7f1ef4b7c640 1 librados: starting objecter 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.556+0000 7f1ef4b7c640 1 Processor -- start 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.556+0000 7f1ef4b7c640 1 -- start start 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.556+0000 7f1ef4b7c640 1 librados: setting wanted keys 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.556+0000 7f1ef4b7c640 1 librados: calling monclient init 2024-03-25T18:27:10.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.556+0000 7f1ef4b7c640 10 monclient: init 2024-03-25T18:27:10.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.557+0000 7f1ef4b7c640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:10.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.557+0000 7f1ef4b7c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:10.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.557+0000 7f1ef4b7c640 10 monclient(hunting): picked mon.noname-c con 0x7f1ef00b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:10.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.557+0000 7f1ef4b7c640 10 monclient(hunting): picked mon.noname-a con 0x7f1ef00b4d60 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:10.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.557+0000 7f1ef4b7c640 10 monclient(hunting): picked mon.noname-b con 0x7f1ef014be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:10.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.557+0000 7f1ef4b7c640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef01406f0 con 0x7f1ef00b4d60 2024-03-25T18:27:10.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.557+0000 7f1ef4b7c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef0140380 con 0x7f1ef014be90 2024-03-25T18:27:10.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.557+0000 7f1ef4b7c640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef0144ce0 con 0x7f1ef00b1730 2024-03-25T18:27:10.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.557+0000 7f1ef4b7c640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:10.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.557+0000 7f1ef4b7c640 10 monclient(hunting): authenticate will time out at 7448.725403s 2024-03-25T18:27:10.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.568+0000 7f1eee59c640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f1ef00b1730 0x7f1ef0131e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:47014/0 (socket says 172.21.15.99:47014) 2024-03-25T18:27:10.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.568+0000 7f1eee59c640 1 -- 172.21.15.99:0/341387217 learned_addr learned my addr 172.21.15.99:0/341387217 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:27:10.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.571+0000 7f1ee6ffd640 10 client.?.objecter ms_handle_connect 0x7f1ef014be90 2024-03-25T18:27:10.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.571+0000 7f1ee6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:10.574 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.573+0000 7f1ee6ffd640 10 client.?.objecter ms_handle_connect 0x7f1ef00b1730 2024-03-25T18:27:10.574 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.573+0000 7f1ee6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:10.574 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.573+0000 7f1ee6ffd640 10 client.?.objecter ms_handle_connect 0x7f1ef00b4d60 2024-03-25T18:27:10.574 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.573+0000 7f1ee6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:10.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.583+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360156899 0 0) 0x7f1ef0144ce0 con 0x7f1ef00b1730 2024-03-25T18:27:10.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.583+0000 7f1ee6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:10.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.583+0000 7f1ee6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:10.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.583+0000 7f1ee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:10.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.583+0000 7f1ee6ffd640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:27:10.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.583+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ed0004eb0 con 0x7f1ef00b1730 2024-03-25T18:27:10.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.584+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203358664 0 0) 0x7f1ef01406f0 con 0x7f1ef00b4d60 2024-03-25T18:27:10.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.584+0000 7f1ee6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:10.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.584+0000 7f1ee6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:10.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.584+0000 7f1ee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:10.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.584+0000 7f1ee6ffd640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:27:10.586 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.584+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef0144ce0 con 0x7f1ef00b4d60 2024-03-25T18:27:10.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.595+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761226830 0 0) 0x7f1ef0140380 con 0x7f1ef014be90 2024-03-25T18:27:10.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.595+0000 7f1ee6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:10.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.595+0000 7f1ee6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:10.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.595+0000 7f1ee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:10.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.595+0000 7f1ee6ffd640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:27:10.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.595+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef01406f0 con 0x7f1ef014be90 2024-03-25T18:27:10.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.597+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 857226707 0 0) 0x7f1ef0144ce0 con 0x7f1ef00b4d60 2024-03-25T18:27:10.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.597+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ef0140380 con 0x7f1ef00b4d60 2024-03-25T18:27:10.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.597+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f1ed8001950 con 0x7f1ef00b4d60 2024-03-25T18:27:10.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.597+0000 7f1ee6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:10.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.597+0000 7f1ee6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:10.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.597+0000 7f1ee6ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:10.598 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:10.598 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:10.598 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:10.598 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:10.599 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:10.599 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:10.599 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:10.599 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:10.599 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:10.599 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:10.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.597+0000 7f1ee6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:10.608 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 727693177 0 0) 0x7f1ef0140380 con 0x7f1ef00b4d60 2024-03-25T18:27:10.608 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 >> v1:172.21.15.99:6790/0 conn(0x7f1ef00b1730 legacy=0x7f1ef0131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:10.608 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 >> v1:172.21.15.190:6789/0 conn(0x7f1ef014be90 legacy=0x7f1ef01d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:10.608 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ee6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ee6ffd640 1 monclient: found mon.a 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ee6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ee6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef0132520 con 0x7f1ef00b4d60 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ee6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ee6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ee6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ef4b7c640 5 monclient: authenticate success, global_id 4302 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ef4b7c640 10 monclient: _renew_subs 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ef4b7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/341387217 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef0132770 con 0x7f1ef00b4d60 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ef4b7c640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.606+0000 7f1ef4b7c640 10 monclient: _renew_subs 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.607+0000 7f1ef4b7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:27:10.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.607+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/341387217 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef013e130 con 0x7f1ef00b4d60 2024-03-25T18:27:10.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.607+0000 7f1ef4b7c640 1 librados: init done 2024-03-25T18:27:10.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.608+0000 7f1ef4b7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:10.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.608+0000 7f1ef4b7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:10.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.608+0000 7f1ef4b7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:27:10.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.608+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/341387217 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef0140510 con 0x7f1ef00b4d60 2024-03-25T18:27:10.921 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.920+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ed8001d60 con 0x7f1ef00b4d60 2024-03-25T18:27:10.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.920+0000 7f1ee6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:10.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.920+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f1ed8004980 con 0x7f1ef00b4d60 2024-03-25T18:27:10.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.920+0000 7f1ee6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:10.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.920+0000 7f1ee6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:10.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.920+0000 7f1ee6ffd640 10 monclient: dump: 2024-03-25T18:27:10.922 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:10.922 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:10.922 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:10.922 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:10.922 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:10.922 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:10.922 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:10.922 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:10.923 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:10.923 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:10.923 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.920+0000 7f1eed59a640 4 set_mon_vals no callback set 2024-03-25T18:27:10.923 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.921+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f1ed8004ba0 con 0x7f1ef00b4d60 2024-03-25T18:27:10.923 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.921+0000 7f1eee59c640 10 monclient: get_auth_request con 0x7f1ed006de20 auth_method 0 2024-03-25T18:27:10.923 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.922+0000 7f1ee6ffd640 10 client.4302.objecter ms_handle_connect 0x7f1ed006de20 2024-03-25T18:27:10.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.926+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f1ed8054740 con 0x7f1ef00b4d60 2024-03-25T18:27:10.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.926+0000 7f1ee6ffd640 10 client.4302.objecter ms_dispatch 0x7f1ef0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:10.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.926+0000 7f1ee6ffd640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:10.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.926+0000 7f1ee6ffd640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:10.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.926+0000 7f1ee6ffd640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-25T18:27:10.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.926+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ed8050080 con 0x7f1ef00b4d60 2024-03-25T18:27:10.928 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.926+0000 7f1ee6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:10.928 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:10.926+0000 7f1ee6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:11.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.181+0000 7f1ef4b7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:11.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.181+0000 7f1ef4b7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:11.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.181+0000 7f1ef4b7c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:27:11.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.181+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/341387217 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ef00627b0 con 0x7f1ef00b4d60 2024-03-25T18:27:11.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.203+0000 7f1ee6ffd640 1 -- 172.21.15.99:0/341387217 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f1ed8052180 con 0x7f1ef00b4d60 2024-03-25T18:27:11.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.203+0000 7f1ee6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:11.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.203+0000 7f1ee6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:11.205 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:27:11.205 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:11.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.206+0000 7f1ef4b7c640 10 librados: watch_flush enter 2024-03-25T18:27:11.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.206+0000 7f1ef4b7c640 10 librados: watch_flush exit 2024-03-25T18:27:11.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.206+0000 7f1ef4b7c640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-25T18:27:11.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.206+0000 7f1ef4b7c640 10 client.4302.objecter successfully canceled tick 2024-03-25T18:27:11.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.206+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/341387217 >> v1:172.21.15.190:6800/4220409699 conn(0x7f1ed006de20 legacy=0x7f1ed00702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.206+0000 7f1ef4b7c640 10 monclient: shutdown 2024-03-25T18:27:11.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.206+0000 7f1ef4b7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:11.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.206+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/341387217 >> v1:172.21.15.99:6789/0 conn(0x7f1ef00b4d60 legacy=0x7f1ef01d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.206+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/341387217 shutdown_connections 2024-03-25T18:27:11.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.206+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/341387217 >> 172.21.15.99:0/341387217 conn(0x7f1ef00617a0 msgr2=0x7f1ef013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:11.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.206+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/341387217 shutdown_connections 2024-03-25T18:27:11.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.206+0000 7f1ef4b7c640 1 -- 172.21.15.99:0/341387217 wait complete. 2024-03-25T18:27:11.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:11.206+0000 7f1ef4b7c640 1 librados: shutdown 2024-03-25T18:27:11.219 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:13.752 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:02:14.344 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:13.761 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:02:14.354 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:13.804 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:02:14.396 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:13.814 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:02:14.373 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:13.818 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:02:14.377 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:13.830 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:02:14.422 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:13.835 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:02:14.428 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:13.843 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:02:14.402 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:13.865 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:02:14.457 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.027 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:18.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.135+0000 7fbd10e43640 10 monclient: get_monmap_and_config 2024-03-25T18:27:18.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.135+0000 7fbd10e43640 10 monclient: build_initial_monmap 2024-03-25T18:27:18.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.135+0000 7fbd10e43640 10 monclient: monmap: 2024-03-25T18:27:18.136 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:27:18.136 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:18.136 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:27:18.136207+0000 2024-03-25T18:27:18.136 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:27:18.136207+0000 2024-03-25T18:27:18.136 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:18.136 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:18.136 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:27:18.137 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:27:18.137 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:27:18.137 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:18.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.135+0000 7fbd10e43640 1 Processor -- start 2024-03-25T18:27:18.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.136+0000 7fbd10e43640 1 -- start start 2024-03-25T18:27:18.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.136+0000 7fbd10e43640 10 monclient: init 2024-03-25T18:27:18.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.136+0000 7fbd10e43640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:18.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.136+0000 7fbd10e43640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:18.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.136+0000 7fbd10e43640 10 monclient(hunting): picked mon.noname-b con 0x7fbd0c0a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:18.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.136+0000 7fbd10e43640 10 monclient(hunting): picked mon.noname-c con 0x7fbd0c13f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:18.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.136+0000 7fbd10e43640 10 monclient(hunting): picked mon.noname-a con 0x7fbd0c1430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:18.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.136+0000 7fbd10e43640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0c142520 con 0x7fbd0c1430c0 2024-03-25T18:27:18.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.136+0000 7fbd10e43640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0c0b28a0 con 0x7fbd0c13f710 2024-03-25T18:27:18.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.136+0000 7fbd10e43640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0c0b2a30 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.136+0000 7fbd10e43640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:18.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.136+0000 7fbd10e43640 10 monclient(hunting): authenticate will time out at 7456.304593s 2024-03-25T18:27:18.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.151+0000 7fbd03fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fbd0c13f710 0x7fbd0c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:49106/0 (socket says 172.21.15.99:49106) 2024-03-25T18:27:18.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.151+0000 7fbd03fff640 1 -- 172.21.15.99:0/2532549673 learned_addr learned my addr 172.21.15.99:0/2532549673 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:27:18.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.166+0000 7fbd09d9b640 1 -- 172.21.15.99:0/2532549673 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176322120 0 0) 0x7fbd0c0b28a0 con 0x7fbd0c13f710 2024-03-25T18:27:18.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.166+0000 7fbd09d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.166+0000 7fbd09d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.166+0000 7fbd09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.166+0000 7fbd09d9b640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:27:18.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.166+0000 7fbd09d9b640 1 -- 172.21.15.99:0/2532549673 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf8003520 con 0x7fbd0c13f710 2024-03-25T18:27:18.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 1 -- 172.21.15.99:0/2532549673 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2335097226 0 0) 0x7fbcf8003520 con 0x7fbd0c13f710 2024-03-25T18:27:18.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 1 -- 172.21.15.99:0/2532549673 >> v1:172.21.15.99:6790/0 conn(0x7fbd0c0a98f0 legacy=0x7fbd0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 1 -- 172.21.15.99:0/2532549673 >> v1:172.21.15.99:6789/0 conn(0x7fbd0c1430c0 legacy=0x7fbd0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:27:18.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 1 monclient: found mon.noname-c 2024-03-25T18:27:18.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:18.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:27:18.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 1 -- 172.21.15.99:0/2532549673 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0c0b44c0 con 0x7fbd0c13f710 2024-03-25T18:27:18.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 10 monclient: _finish_auth 0 2024-03-25T18:27:18.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:27:18.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:18.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 1 -- 172.21.15.99:0/2532549673 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbcf4001690 con 0x7fbd0c13f710 2024-03-25T18:27:18.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:18.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:18.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd09d9b640 10 monclient: dump: 2024-03-25T18:27:18.179 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:18.179 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:18.180 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:18.180 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:18.180 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:18.180 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:18.180 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:18.180 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:18.180 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:18.180 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:18.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd10e43640 5 monclient: authenticate success, global_id 4315 2024-03-25T18:27:18.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.175+0000 7fbd10e43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:18.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.185+0000 7fbd09d9b640 1 -- 172.21.15.99:0/2532549673 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcf40017f0 con 0x7fbd0c13f710 2024-03-25T18:27:18.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.185+0000 7fbd09d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:18.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.185+0000 7fbd09d9b640 1 -- 172.21.15.99:0/2532549673 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbcf4001c40 con 0x7fbd0c13f710 2024-03-25T18:27:18.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.185+0000 7fbd09d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:18.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.185+0000 7fbd09d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:18.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.185+0000 7fbd09d9b640 10 monclient: dump: 2024-03-25T18:27:18.186 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:18.186 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:18.186 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:18.186 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:18.186 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:18.186 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:18.186 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:18.187 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:18.187 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:18.187 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:18.187 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.185+0000 7fbd10e43640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:18.187 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.185+0000 7fbd10e43640 4 set_mon_vals no callback set 2024-03-25T18:27:18.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.185+0000 7fbd10e43640 10 monclient: shutdown 2024-03-25T18:27:18.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.185+0000 7fbd10e43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:18.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.185+0000 7fbd10e43640 1 -- 172.21.15.99:0/2532549673 >> v1:172.21.15.190:6789/0 conn(0x7fbd0c13f710 legacy=0x7fbd0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.186+0000 7fbd10e43640 1 -- 172.21.15.99:0/2532549673 shutdown_connections 2024-03-25T18:27:18.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.186+0000 7fbd10e43640 1 -- 172.21.15.99:0/2532549673 >> 172.21.15.99:0/2532549673 conn(0x7fbd0c0b7ea0 msgr2=0x7fbd0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:18.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.186+0000 7fbd10e43640 1 -- 172.21.15.99:0/2532549673 shutdown_connections 2024-03-25T18:27:18.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.186+0000 7fbd10e43640 1 -- 172.21.15.99:0/2532549673 wait complete. 2024-03-25T18:27:18.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.187+0000 7fbd10e43640 10 monclient: build_initial_monmap 2024-03-25T18:27:18.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.187+0000 7fbd10e43640 10 monclient: monmap: 2024-03-25T18:27:18.189 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:27:18.189 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:18.189 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:27:18.189 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:27:18.189 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:18.189 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:27:18.189 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:27:18.189 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:27:18.189 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:27:18.190 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:18.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.187+0000 7fbd10e43640 1 librados: starting msgr at 2024-03-25T18:27:18.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.187+0000 7fbd10e43640 1 librados: starting objecter 2024-03-25T18:27:18.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.187+0000 7fbd10e43640 1 Processor -- start 2024-03-25T18:27:18.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.187+0000 7fbd10e43640 1 -- start start 2024-03-25T18:27:18.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.187+0000 7fbd10e43640 1 librados: setting wanted keys 2024-03-25T18:27:18.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.187+0000 7fbd10e43640 1 librados: calling monclient init 2024-03-25T18:27:18.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.187+0000 7fbd10e43640 10 monclient: init 2024-03-25T18:27:18.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.188+0000 7fbd10e43640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:18.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.188+0000 7fbd10e43640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:18.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.188+0000 7fbd10e43640 10 monclient(hunting): picked mon.noname-c con 0x7fbd0c0a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:18.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.188+0000 7fbd10e43640 10 monclient(hunting): picked mon.noname-b con 0x7fbd0c13f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:18.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.188+0000 7fbd10e43640 10 monclient(hunting): picked mon.noname-a con 0x7fbd0c1430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:18.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.188+0000 7fbd10e43640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0c138b20 con 0x7fbd0c1430c0 2024-03-25T18:27:18.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.188+0000 7fbd10e43640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0c137a80 con 0x7fbd0c13f710 2024-03-25T18:27:18.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.188+0000 7fbd10e43640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0c139880 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.188+0000 7fbd10e43640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:18.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.188+0000 7fbd10e43640 10 monclient(hunting): authenticate will time out at 7456.356483s 2024-03-25T18:27:18.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.194+0000 7fbd0a59c640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7fbd0c0a98f0 0x7fbd0c137370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:40024/0 (socket says 172.21.15.99:40024) 2024-03-25T18:27:18.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.194+0000 7fbd0a59c640 1 -- 172.21.15.99:0/184764820 learned_addr learned my addr 172.21.15.99:0/184764820 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:27:18.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.197+0000 7fbd02ffd640 10 client.?.objecter ms_handle_connect 0x7fbd0c13f710 2024-03-25T18:27:18.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.197+0000 7fbd02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:18.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.198+0000 7fbd02ffd640 10 client.?.objecter ms_handle_connect 0x7fbd0c0a98f0 2024-03-25T18:27:18.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.198+0000 7fbd02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:18.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.199+0000 7fbd02ffd640 10 client.?.objecter ms_handle_connect 0x7fbd0c1430c0 2024-03-25T18:27:18.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.199+0000 7fbd02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:18.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.206+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038908597 0 0) 0x7fbd0c137a80 con 0x7fbd0c13f710 2024-03-25T18:27:18.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.206+0000 7fbd02ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.206+0000 7fbd02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.206+0000 7fbd02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.206+0000 7fbd02ffd640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:27:18.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.206+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcec004eb0 con 0x7fbd0c13f710 2024-03-25T18:27:18.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.207+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802798492 0 0) 0x7fbd0c139880 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.207+0000 7fbd02ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.207+0000 7fbd02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.207+0000 7fbd02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.207+0000 7fbd02ffd640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:27:18.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.207+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd0c137a80 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.215+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282103981 0 0) 0x7fbd0c138b20 con 0x7fbd0c1430c0 2024-03-25T18:27:18.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.215+0000 7fbd02ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.215+0000 7fbd02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.215+0000 7fbd02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.215+0000 7fbd02ffd640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:27:18.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.215+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd0c139880 con 0x7fbd0c1430c0 2024-03-25T18:27:18.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.230+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3855755960 0 0) 0x7fbd0c137a80 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.230+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd0c138b20 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.230+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbcfc0019b0 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.230+0000 7fbd02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:18.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.230+0000 7fbd02ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:18.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.230+0000 7fbd02ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:18.232 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:18.232 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:18.232 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:18.232 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:18.232 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:18.232 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:18.232 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:18.232 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:18.232 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:18.232 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:18.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.230+0000 7fbd02ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:18.241 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.240+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1162730369 0 0) 0x7fbcec004eb0 con 0x7fbd0c13f710 2024-03-25T18:27:18.241 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.240+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd0c137a80 con 0x7fbd0c13f710 2024-03-25T18:27:18.241 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.240+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbcf4001b00 con 0x7fbd0c13f710 2024-03-25T18:27:18.241 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.240+0000 7fbd02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:18.241 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.240+0000 7fbd02ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:18.241 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.240+0000 7fbd02ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:18.241 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:18.241 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:18.241 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:18.241 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:18.241 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:18.242 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:18.242 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:18.242 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:18.242 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:18.242 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:18.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.242+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3035532343 0 0) 0x7fbd0c139880 con 0x7fbd0c1430c0 2024-03-25T18:27:18.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.242+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcec004eb0 con 0x7fbd0c1430c0 2024-03-25T18:27:18.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.242+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbd04001a60 con 0x7fbd0c1430c0 2024-03-25T18:27:18.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.242+0000 7fbd02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:18.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.242+0000 7fbd02ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:18.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.242+0000 7fbd02ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:18.244 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:18.244 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:18.244 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:18.244 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:18.245 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:18.245 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:18.245 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:18.245 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:18.245 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:18.245 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:18.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3054233057 0 0) 0x7fbd0c138b20 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 >> v1:172.21.15.190:6789/0 conn(0x7fbd0c13f710 legacy=0x7fbd0c139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 >> v1:172.21.15.99:6789/0 conn(0x7fbd0c1430c0 legacy=0x7fbd0c1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd02ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:18.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd02ffd640 1 monclient: found mon.c 2024-03-25T18:27:18.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:18.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd02ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:18.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0c139a10 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd02ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:18.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd02ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:18.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:18.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd10e43640 5 monclient: authenticate success, global_id 4319 2024-03-25T18:27:18.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd10e43640 10 monclient: _renew_subs 2024-03-25T18:27:18.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd10e43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:18.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd10e43640 1 -- 172.21.15.99:0/184764820 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0c1dab00 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd10e43640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:18.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd10e43640 10 monclient: _renew_subs 2024-03-25T18:27:18.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd10e43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:18.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd10e43640 1 -- 172.21.15.99:0/184764820 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0c13b4f0 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.248+0000 7fbd10e43640 1 librados: init done 2024-03-25T18:27:18.252 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.249+0000 7fbd10e43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:18.252 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.249+0000 7fbd10e43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:18.252 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.249+0000 7fbd10e43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:18.252 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.249+0000 7fbd10e43640 1 -- 172.21.15.99:0/184764820 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0c074cc0 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.548+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcfc003610 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.548+0000 7fbd02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:18.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.548+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbcfc0039a0 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.548+0000 7fbd02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:18.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.548+0000 7fbd02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:18.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.548+0000 7fbd02ffd640 10 monclient: dump: 2024-03-25T18:27:18.550 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:18.550 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:18.550 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:18.550 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:18.550 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:18.550 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:18.550 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:18.550 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:18.550 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:18.551 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:18.551 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.548+0000 7fbd0959a640 4 set_mon_vals no callback set 2024-03-25T18:27:18.552 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.549+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fbcfc003bc0 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.552 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.550+0000 7fbd03fff640 10 monclient: get_auth_request con 0x7fbcec06e0f0 auth_method 0 2024-03-25T18:27:18.552 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.550+0000 7fbd02ffd640 10 client.4319.objecter ms_handle_connect 0x7fbcec06e0f0 2024-03-25T18:27:18.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.558+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fbcfc0547c0 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.558+0000 7fbd02ffd640 10 client.4319.objecter ms_dispatch 0x7fbd0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:18.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.558+0000 7fbd02ffd640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:18.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.558+0000 7fbd02ffd640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:18.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.558+0000 7fbd02ffd640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-25T18:27:18.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.558+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbcfc0502d0 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.558+0000 7fbd02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:18.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.558+0000 7fbd02ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:18.798 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.797+0000 7fbd10e43640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:18.798 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.797+0000 7fbd10e43640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:18.798 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.797+0000 7fbd10e43640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:18.798 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.797+0000 7fbd10e43640 1 -- 172.21.15.99:0/184764820 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd0c1dae00 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.824+0000 7fbd02ffd640 1 -- 172.21.15.99:0/184764820 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fbcfc052200 con 0x7fbd0c0a98f0 2024-03-25T18:27:18.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.824+0000 7fbd02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:18.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.824+0000 7fbd02ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:18.825 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:27:18.826 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:18.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.826+0000 7fbd10e43640 10 librados: watch_flush enter 2024-03-25T18:27:18.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.826+0000 7fbd10e43640 10 librados: watch_flush exit 2024-03-25T18:27:18.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.826+0000 7fbd10e43640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-25T18:27:18.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.826+0000 7fbd10e43640 10 client.4319.objecter successfully canceled tick 2024-03-25T18:27:18.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.826+0000 7fbd10e43640 1 -- 172.21.15.99:0/184764820 >> v1:172.21.15.190:6800/4220409699 conn(0x7fbcec06e0f0 legacy=0x7fbcec0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.826+0000 7fbd10e43640 10 monclient: shutdown 2024-03-25T18:27:18.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.826+0000 7fbd10e43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:18.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.826+0000 7fbd10e43640 1 -- 172.21.15.99:0/184764820 >> v1:172.21.15.99:6790/0 conn(0x7fbd0c0a98f0 legacy=0x7fbd0c137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.826+0000 7fbd10e43640 1 -- 172.21.15.99:0/184764820 shutdown_connections 2024-03-25T18:27:18.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.826+0000 7fbd10e43640 1 -- 172.21.15.99:0/184764820 >> 172.21.15.99:0/184764820 conn(0x7fbd0c0b7ea0 msgr2=0x7fbd0c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:18.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.826+0000 7fbd10e43640 1 -- 172.21.15.99:0/184764820 shutdown_connections 2024-03-25T18:27:18.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.826+0000 7fbd10e43640 1 -- 172.21.15.99:0/184764820 wait complete. 2024-03-25T18:27:18.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:18.826+0000 7fbd10e43640 1 librados: shutdown 2024-03-25T18:27:18.840 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:25.646 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:25.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.759+0000 7f2aea1c8640 10 monclient: get_monmap_and_config 2024-03-25T18:27:25.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.759+0000 7f2aea1c8640 10 monclient: build_initial_monmap 2024-03-25T18:27:25.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.759+0000 7f2aea1c8640 10 monclient: monmap: 2024-03-25T18:27:25.762 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:27:25.762 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:25.762 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:27:25.760880+0000 2024-03-25T18:27:25.762 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:27:25.760880+0000 2024-03-25T18:27:25.762 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:25.762 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:25.762 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:27:25.762 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:27:25.763 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:27:25.763 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:25.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.760+0000 7f2aea1c8640 1 Processor -- start 2024-03-25T18:27:25.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.760+0000 7f2aea1c8640 1 -- start start 2024-03-25T18:27:25.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.760+0000 7f2aea1c8640 10 monclient: init 2024-03-25T18:27:25.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.761+0000 7f2aea1c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:25.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.761+0000 7f2aea1c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:25.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.761+0000 7f2aea1c8640 10 monclient(hunting): picked mon.noname-c con 0x7f2ae40b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:25.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.761+0000 7f2aea1c8640 10 monclient(hunting): picked mon.noname-b con 0x7f2ae40b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:25.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.761+0000 7f2aea1c8640 10 monclient(hunting): picked mon.noname-a con 0x7f2ae414be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:25.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.761+0000 7f2aea1c8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae40af940 con 0x7f2ae414be90 2024-03-25T18:27:25.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.761+0000 7f2aea1c8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae40b0a70 con 0x7f2ae40b4d60 2024-03-25T18:27:25.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.761+0000 7f2aea1c8640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae40b0d50 con 0x7f2ae40b1730 2024-03-25T18:27:25.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.761+0000 7f2aea1c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:25.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.761+0000 7f2aea1c8640 10 monclient(hunting): authenticate will time out at 7463.929540s 2024-03-25T18:27:25.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.775+0000 7f2ae3fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2ae414be90 0x7f2ae414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51182/0 (socket says 172.21.15.99:51182) 2024-03-25T18:27:25.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.775+0000 7f2ae3fff640 1 -- 172.21.15.99:0/3388804767 learned_addr learned my addr 172.21.15.99:0/3388804767 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:27:25.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.782+0000 7f2ae27fc640 1 -- 172.21.15.99:0/3388804767 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067550213 0 0) 0x7f2ae40af940 con 0x7f2ae414be90 2024-03-25T18:27:25.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.782+0000 7f2ae27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:25.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.782+0000 7f2ae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:25.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.782+0000 7f2ae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:25.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.782+0000 7f2ae27fc640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:27:25.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.782+0000 7f2ae27fc640 1 -- 172.21.15.99:0/3388804767 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2acc003520 con 0x7f2ae414be90 2024-03-25T18:27:25.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 1 -- 172.21.15.99:0/3388804767 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3946330523 0 0) 0x7f2acc003520 con 0x7f2ae414be90 2024-03-25T18:27:25.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 1 -- 172.21.15.99:0/3388804767 >> v1:172.21.15.99:6790/0 conn(0x7f2ae40b1730 legacy=0x7f2ae40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:25.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 1 -- 172.21.15.99:0/3388804767 >> v1:172.21.15.190:6789/0 conn(0x7f2ae40b4d60 legacy=0x7f2ae413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:25.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:25.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 1 monclient: found mon.noname-a 2024-03-25T18:27:25.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:25.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:27:25.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 1 -- 172.21.15.99:0/3388804767 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae40b1030 con 0x7f2ae414be90 2024-03-25T18:27:25.795 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:25.795 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:25.795 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:25.795 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 1 -- 172.21.15.99:0/3388804767 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2adc001a10 con 0x7f2ae414be90 2024-03-25T18:27:25.795 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:25.795 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:25.795 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2ae27fc640 10 monclient: dump: 2024-03-25T18:27:25.795 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:25.795 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:25.795 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:25.795 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:25.795 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:25.795 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:25.795 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:25.796 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:25.796 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:25.796 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:25.796 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2aea1c8640 5 monclient: authenticate success, global_id 4311 2024-03-25T18:27:25.796 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.793+0000 7f2aea1c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:25.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.824+0000 7f2ae27fc640 1 -- 172.21.15.99:0/3388804767 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2adc001b70 con 0x7f2ae414be90 2024-03-25T18:27:25.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.824+0000 7f2ae27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:25.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.824+0000 7f2ae27fc640 1 -- 172.21.15.99:0/3388804767 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2adc001f80 con 0x7f2ae414be90 2024-03-25T18:27:25.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.824+0000 7f2aea1c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:25.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.824+0000 7f2aea1c8640 4 set_mon_vals no callback set 2024-03-25T18:27:25.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.824+0000 7f2ae27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:25.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.824+0000 7f2ae27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:25.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.824+0000 7f2ae27fc640 10 monclient: dump: 2024-03-25T18:27:25.825 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:25.825 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:25.825 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:25.826 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:25.826 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:25.826 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:25.826 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:25.826 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:25.826 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:25.826 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:25.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.825+0000 7f2aea1c8640 10 monclient: shutdown 2024-03-25T18:27:25.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.825+0000 7f2aea1c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:25.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.825+0000 7f2aea1c8640 1 -- 172.21.15.99:0/3388804767 >> v1:172.21.15.99:6789/0 conn(0x7f2ae414be90 legacy=0x7f2ae414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:25.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.825+0000 7f2aea1c8640 1 -- 172.21.15.99:0/3388804767 shutdown_connections 2024-03-25T18:27:25.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.825+0000 7f2aea1c8640 1 -- 172.21.15.99:0/3388804767 >> 172.21.15.99:0/3388804767 conn(0x7f2ae40617a0 msgr2=0x7f2ae4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.825+0000 7f2aea1c8640 1 -- 172.21.15.99:0/3388804767 shutdown_connections 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.825+0000 7f2aea1c8640 1 -- 172.21.15.99:0/3388804767 wait complete. 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.825+0000 7f2aea1c8640 10 monclient: build_initial_monmap 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.825+0000 7f2aea1c8640 10 monclient: monmap: 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:25.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.826+0000 7f2aea1c8640 1 librados: starting msgr at 2024-03-25T18:27:25.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.826+0000 7f2aea1c8640 1 librados: starting objecter 2024-03-25T18:27:25.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.826+0000 7f2aea1c8640 1 Processor -- start 2024-03-25T18:27:25.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.826+0000 7f2aea1c8640 1 -- start start 2024-03-25T18:27:25.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.826+0000 7f2aea1c8640 1 librados: setting wanted keys 2024-03-25T18:27:25.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.826+0000 7f2aea1c8640 1 librados: calling monclient init 2024-03-25T18:27:25.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.826+0000 7f2aea1c8640 10 monclient: init 2024-03-25T18:27:25.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.826+0000 7f2aea1c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:25.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.827+0000 7f2aea1c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:25.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.827+0000 7f2aea1c8640 10 monclient(hunting): picked mon.noname-b con 0x7f2ae40b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:25.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.827+0000 7f2aea1c8640 10 monclient(hunting): picked mon.noname-a con 0x7f2ae40b4d60 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:25.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.827+0000 7f2aea1c8640 10 monclient(hunting): picked mon.noname-c con 0x7f2ae414be90 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:25.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.827+0000 7f2aea1c8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae4137e30 con 0x7f2ae40b4d60 2024-03-25T18:27:25.831 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.827+0000 7f2aea1c8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae4137ae0 con 0x7f2ae40b1730 2024-03-25T18:27:25.831 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.827+0000 7f2aea1c8640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae4145290 con 0x7f2ae414be90 2024-03-25T18:27:25.831 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.827+0000 7f2aea1c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:25.831 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.827+0000 7f2aea1c8640 10 monclient(hunting): authenticate will time out at 7463.994954s 2024-03-25T18:27:25.833 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.833+0000 7f2ae3fff640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f2ae414be90 0x7f2ae41e22d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:50100/0 (socket says 172.21.15.99:50100) 2024-03-25T18:27:25.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.833+0000 7f2ae3fff640 1 -- 172.21.15.99:0/1198904777 learned_addr learned my addr 172.21.15.99:0/1198904777 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:27:25.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.834+0000 7f2acbfff640 10 client.?.objecter ms_handle_connect 0x7f2ae40b1730 2024-03-25T18:27:25.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.834+0000 7f2acbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:25.837 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.836+0000 7f2acbfff640 10 client.?.objecter ms_handle_connect 0x7f2ae414be90 2024-03-25T18:27:25.837 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.836+0000 7f2acbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:25.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.842+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435690206 0 0) 0x7f2ae4137ae0 con 0x7f2ae40b1730 2024-03-25T18:27:25.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.842+0000 7f2acbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:25.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.842+0000 7f2acbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:25.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.842+0000 7f2acbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:25.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.842+0000 7f2acbfff640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:27:25.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.842+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad0004eb0 con 0x7f2ae40b1730 2024-03-25T18:27:25.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.843+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6932926 0 0) 0x7f2ae4145290 con 0x7f2ae414be90 2024-03-25T18:27:25.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.843+0000 7f2acbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:25.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.843+0000 7f2acbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:25.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.843+0000 7f2acbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:25.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.843+0000 7f2acbfff640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:27:25.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.843+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae4137ae0 con 0x7f2ae414be90 2024-03-25T18:27:25.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.846+0000 7f2acbfff640 10 client.?.objecter ms_handle_connect 0x7f2ae40b4d60 2024-03-25T18:27:25.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.846+0000 7f2acbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:25.869 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.868+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2584124254 0 0) 0x7f2ae4137ae0 con 0x7f2ae414be90 2024-03-25T18:27:25.869 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.868+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ae4145290 con 0x7f2ae414be90 2024-03-25T18:27:25.869 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.868+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2adc001f40 con 0x7f2ae414be90 2024-03-25T18:27:25.869 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.868+0000 7f2acbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:25.869 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.868+0000 7f2acbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:25.869 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.868+0000 7f2acbfff640 10 monclient(hunting): dump: 2024-03-25T18:27:25.869 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:25.870 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:25.870 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:25.870 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:25.870 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:25.870 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:25.870 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:25.870 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:25.870 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:25.870 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:25.870 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.868+0000 7f2acbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:25.928 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1609573532 0 0) 0x7f2ae4145290 con 0x7f2ae414be90 2024-03-25T18:27:25.928 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 >> v1:172.21.15.190:6789/0 conn(0x7f2ae40b1730 legacy=0x7f2ae4140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:25.929 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 >> v1:172.21.15.99:6789/0 conn(0x7f2ae40b4d60 legacy=0x7f2ae41d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:25.929 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2acbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:25.929 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2acbfff640 1 monclient: found mon.c 2024-03-25T18:27:25.929 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2acbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:25.929 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2acbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:25.930 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae4145420 con 0x7f2ae414be90 2024-03-25T18:27:25.930 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2acbfff640 10 monclient: _finish_auth 0 2024-03-25T18:27:25.930 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2acbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:25.930 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2acbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:25.930 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2aea1c8640 5 monclient: authenticate success, global_id 4325 2024-03-25T18:27:25.930 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2aea1c8640 10 monclient: _renew_subs 2024-03-25T18:27:25.931 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2aea1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:25.931 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2aea1c8640 1 -- 172.21.15.99:0/1198904777 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae41d7f40 con 0x7f2ae414be90 2024-03-25T18:27:25.931 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2aea1c8640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:25.931 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2aea1c8640 10 monclient: _renew_subs 2024-03-25T18:27:25.931 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2aea1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:25.931 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2aea1c8640 1 -- 172.21.15.99:0/1198904777 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae41e39f0 con 0x7f2ae414be90 2024-03-25T18:27:25.931 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.926+0000 7f2aea1c8640 1 librados: init done 2024-03-25T18:27:25.932 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.927+0000 7f2aea1c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:25.932 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.927+0000 7f2aea1c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:25.932 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.927+0000 7f2aea1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:25.932 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:25.927+0000 7f2aea1c8640 1 -- 172.21.15.99:0/1198904777 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae4137c70 con 0x7f2ae414be90 2024-03-25T18:27:26.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.253+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2adc004e10 con 0x7f2ae414be90 2024-03-25T18:27:26.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.253+0000 7f2acbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:26.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.253+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2adc005120 con 0x7f2ae414be90 2024-03-25T18:27:26.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.253+0000 7f2acbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:26.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.253+0000 7f2acbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:26.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.253+0000 7f2acbfff640 10 monclient: dump: 2024-03-25T18:27:26.254 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:26.255 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:26.255 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:26.255 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:26.255 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:26.255 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:26.255 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:26.255 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:26.255 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:26.255 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:26.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.253+0000 7f2ae1ffb640 4 set_mon_vals no callback set 2024-03-25T18:27:26.256 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.254+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f2adc01b0f0 con 0x7f2ae414be90 2024-03-25T18:27:26.256 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.254+0000 7f2ae37fe640 10 monclient: get_auth_request con 0x7f2ad006de20 auth_method 0 2024-03-25T18:27:26.256 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.254+0000 7f2acbfff640 10 client.4325.objecter ms_handle_connect 0x7f2ad006de20 2024-03-25T18:27:26.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.259+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f2adc054ae0 con 0x7f2ae414be90 2024-03-25T18:27:26.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.259+0000 7f2acbfff640 10 client.4325.objecter ms_dispatch 0x7f2ae4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:26.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.259+0000 7f2acbfff640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:26.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.259+0000 7f2acbfff640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:26.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.259+0000 7f2acbfff640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-25T18:27:26.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.259+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2adc0505f0 con 0x7f2ae414be90 2024-03-25T18:27:26.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.259+0000 7f2acbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:26.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.259+0000 7f2acbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:26.523 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.522+0000 7f2aea1c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:26.523 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.522+0000 7f2aea1c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:26.523 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.522+0000 7f2aea1c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:27:26.523 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.522+0000 7f2aea1c8640 1 -- 172.21.15.99:0/1198904777 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ae4142220 con 0x7f2ae414be90 2024-03-25T18:27:26.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.559+0000 7f2acbfff640 1 -- 172.21.15.99:0/1198904777 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f2adc052520 con 0x7f2ae414be90 2024-03-25T18:27:26.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.559+0000 7f2acbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:26.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.559+0000 7f2acbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:26.560 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:27:26.560 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:26.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.561+0000 7f2aea1c8640 10 librados: watch_flush enter 2024-03-25T18:27:26.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.561+0000 7f2aea1c8640 10 librados: watch_flush exit 2024-03-25T18:27:26.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.561+0000 7f2aea1c8640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-25T18:27:26.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.561+0000 7f2aea1c8640 10 client.4325.objecter successfully canceled tick 2024-03-25T18:27:26.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.561+0000 7f2aea1c8640 1 -- 172.21.15.99:0/1198904777 >> v1:172.21.15.190:6800/4220409699 conn(0x7f2ad006de20 legacy=0x7f2ad00702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.561+0000 7f2aea1c8640 10 monclient: shutdown 2024-03-25T18:27:26.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.561+0000 7f2aea1c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:26.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.561+0000 7f2aea1c8640 1 -- 172.21.15.99:0/1198904777 >> v1:172.21.15.99:6790/0 conn(0x7f2ae414be90 legacy=0x7f2ae41e22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.561+0000 7f2aea1c8640 1 -- 172.21.15.99:0/1198904777 shutdown_connections 2024-03-25T18:27:26.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.561+0000 7f2aea1c8640 1 -- 172.21.15.99:0/1198904777 >> 172.21.15.99:0/1198904777 conn(0x7f2ae40617a0 msgr2=0x7f2ae413c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:26.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.561+0000 7f2aea1c8640 1 -- 172.21.15.99:0/1198904777 shutdown_connections 2024-03-25T18:27:26.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.561+0000 7f2aea1c8640 1 -- 172.21.15.99:0/1198904777 wait complete. 2024-03-25T18:27:26.564 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:26.561+0000 7f2aea1c8640 1 librados: shutdown 2024-03-25T18:27:26.575 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:33.381 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:33.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.468+0000 7fa028833640 10 monclient: get_monmap_and_config 2024-03-25T18:27:33.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.468+0000 7fa028833640 10 monclient: build_initial_monmap 2024-03-25T18:27:33.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.468+0000 7fa028833640 10 monclient: monmap: 2024-03-25T18:27:33.469 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:27:33.469 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:33.470 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:27:33.469533+0000 2024-03-25T18:27:33.470 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:27:33.469533+0000 2024-03-25T18:27:33.470 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:33.470 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:33.470 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:27:33.470 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:27:33.470 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:27:33.470 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:33.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.469+0000 7fa028833640 1 Processor -- start 2024-03-25T18:27:33.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.469+0000 7fa028833640 1 -- start start 2024-03-25T18:27:33.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.469+0000 7fa028833640 10 monclient: init 2024-03-25T18:27:33.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.469+0000 7fa028833640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:33.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.469+0000 7fa028833640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:33.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.469+0000 7fa028833640 10 monclient(hunting): picked mon.noname-a con 0x7fa0240b4d60 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:33.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.469+0000 7fa028833640 10 monclient(hunting): picked mon.noname-b con 0x7fa0240b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:33.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.469+0000 7fa028833640 10 monclient(hunting): picked mon.noname-c con 0x7fa02414be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:33.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.469+0000 7fa028833640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0240af940 con 0x7fa0240b4d60 2024-03-25T18:27:33.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.469+0000 7fa028833640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0240b0a70 con 0x7fa02414be90 2024-03-25T18:27:33.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.469+0000 7fa028833640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0240b0d50 con 0x7fa0240b1730 2024-03-25T18:27:33.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.469+0000 7fa028833640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:33.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.469+0000 7fa028833640 10 monclient(hunting): authenticate will time out at 7471.637778s 2024-03-25T18:27:33.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.470+0000 7fa02259c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa0240b4d60 0x7fa02413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:54024/0 (socket says 172.21.15.99:54024) 2024-03-25T18:27:33.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:33.470+0000 7fa02259c640 1 -- 172.21.15.99:0/3870280158 learned_addr learned my addr 172.21.15.99:0/3870280158 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:27:36.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.469+0000 7fa020d99640 10 monclient(hunting): tick 2024-03-25T18:27:36.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.469+0000 7fa020d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:27:36.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.469+0000 7fa020d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:27:36.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.469+0000 7fa020d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:27:36.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.469+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 >> v1:172.21.15.99:6790/0 conn(0x7fa0240b1730 legacy=0x7fa0240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:27:36.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.469+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 >> v1:172.21.15.190:6789/0 conn(0x7fa02414be90 legacy=0x7fa02414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:36.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.469+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 >> v1:172.21.15.99:6789/0 conn(0x7fa0240b4d60 legacy=0x7fa02413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:36.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.469+0000 7fa020d99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:36.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.470+0000 7fa020d99640 10 monclient(hunting): picked mon.noname-a con 0x7fa0040036c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:36.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.470+0000 7fa020d99640 10 monclient(hunting): picked mon.noname-b con 0x7fa0240b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:36.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.470+0000 7fa020d99640 10 monclient(hunting): picked mon.noname-c con 0x7fa02414be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:36.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0240b1030 con 0x7fa0040036c0 2024-03-25T18:27:36.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0240b0d50 con 0x7fa02414be90 2024-03-25T18:27:36.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0040103f0 con 0x7fa0240b1730 2024-03-25T18:27:36.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:36.470+0000 7fa020d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:39.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 10 monclient(hunting): tick 2024-03-25T18:27:39.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:27:39.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:27:39.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:27:39.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 >> v1:172.21.15.99:6790/0 conn(0x7fa0240b1730 legacy=0x7fa0040093c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:27:39.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 >> v1:172.21.15.190:6789/0 conn(0x7fa02414be90 legacy=0x7fa00400cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:39.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 >> v1:172.21.15.99:6789/0 conn(0x7fa0040036c0 legacy=0x7fa004005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:39.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:39.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 10 monclient(hunting): picked mon.noname-b con 0x7fa004010730 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:39.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 10 monclient(hunting): picked mon.noname-c con 0x7fa02414be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:39.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 10 monclient(hunting): picked mon.noname-a con 0x7fa0040036c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:39.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa004011650 con 0x7fa0040036c0 2024-03-25T18:27:39.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0040103f0 con 0x7fa02414be90 2024-03-25T18:27:39.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa00400f3a0 con 0x7fa004010730 2024-03-25T18:27:39.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:39.470+0000 7fa020d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:42.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 10 monclient(hunting): tick 2024-03-25T18:27:42.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:27:42.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:27:42.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:27:42.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 >> v1:172.21.15.99:6790/0 conn(0x7fa004010730 legacy=0x7fa004010b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:27:42.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 >> v1:172.21.15.190:6789/0 conn(0x7fa02414be90 legacy=0x7fa00400cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:42.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 >> v1:172.21.15.99:6789/0 conn(0x7fa0040036c0 legacy=0x7fa004005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:42.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:42.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 10 monclient(hunting): picked mon.noname-c con 0x7fa00400f530 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:42.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 10 monclient(hunting): picked mon.noname-a con 0x7fa004009af0 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:42.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 10 monclient(hunting): picked mon.noname-b con 0x7fa004010730 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:42.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0040093d0 con 0x7fa004009af0 2024-03-25T18:27:42.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa00400f3a0 con 0x7fa00400f530 2024-03-25T18:27:42.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 1 -- 172.21.15.99:0/3870280158 --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0040097a0 con 0x7fa004010730 2024-03-25T18:27:42.472 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:42.470+0000 7fa020d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:45.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.372+0000 7fa021d9b640 1 -- 172.21.15.99:0/3870280158 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089159719 0 0) 0x7fa00400f3a0 con 0x7fa00400f530 2024-03-25T18:27:45.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.372+0000 7fa021d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:27:45.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.372+0000 7fa021d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:45.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.372+0000 7fa021d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:45.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.372+0000 7fa021d9b640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:27:45.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.372+0000 7fa021d9b640 1 -- 172.21.15.99:0/3870280158 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa010003520 con 0x7fa00400f530 2024-03-25T18:27:45.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 1 -- 172.21.15.99:0/3870280158 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4126297551 0 0) 0x7fa010003520 con 0x7fa00400f530 2024-03-25T18:27:45.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 1 -- 172.21.15.99:0/3870280158 >> v1:172.21.15.99:6790/0 conn(0x7fa004010730 legacy=0x7fa004010b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:27:45.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 1 -- 172.21.15.99:0/3870280158 >> v1:172.21.15.99:6789/0 conn(0x7fa004009af0 legacy=0x7fa00400cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:45.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:27:45.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 1 monclient: found mon.noname-c 2024-03-25T18:27:45.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:45.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:27:45.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 1 -- 172.21.15.99:0/3870280158 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa004005ae0 con 0x7fa00400f530 2024-03-25T18:27:45.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 10 monclient: _finish_auth 0 2024-03-25T18:27:45.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:27:45.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:45.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 1 -- 172.21.15.99:0/3870280158 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa0140015f0 con 0x7fa00400f530 2024-03-25T18:27:45.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa028833640 5 monclient: authenticate success, global_id 4327 2024-03-25T18:27:45.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa028833640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:45.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:45.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:45.391 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa021d9b640 10 monclient: dump: 2024-03-25T18:27:45.391 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:45.391 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:45.391 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:45.391 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:45.391 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:45.391 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:45.391 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:45.391 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:45.391 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:45.391 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:45.391 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.388+0000 7fa028833640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:45.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.406+0000 7fa021d9b640 1 -- 172.21.15.99:0/3870280158 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa014001750 con 0x7fa00400f530 2024-03-25T18:27:45.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.406+0000 7fa021d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:45.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.406+0000 7fa021d9b640 1 -- 172.21.15.99:0/3870280158 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa014001ba0 con 0x7fa00400f530 2024-03-25T18:27:45.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.406+0000 7fa021d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:45.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.406+0000 7fa021d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:45.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.406+0000 7fa021d9b640 10 monclient: dump: 2024-03-25T18:27:45.407 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:45.407 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:45.407 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:45.408 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:45.408 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:45.408 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:45.408 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:45.408 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:45.408 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:45.408 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:45.408 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.406+0000 7fa028833640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:45.408 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.406+0000 7fa028833640 4 set_mon_vals no callback set 2024-03-25T18:27:45.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.407+0000 7fa028833640 10 monclient: shutdown 2024-03-25T18:27:45.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.407+0000 7fa028833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:45.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.407+0000 7fa028833640 1 -- 172.21.15.99:0/3870280158 >> v1:172.21.15.190:6789/0 conn(0x7fa00400f530 legacy=0x7fa00400f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:45.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.407+0000 7fa028833640 1 -- 172.21.15.99:0/3870280158 shutdown_connections 2024-03-25T18:27:45.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.407+0000 7fa028833640 1 -- 172.21.15.99:0/3870280158 >> 172.21.15.99:0/3870280158 conn(0x7fa0240617a0 msgr2=0x7fa024061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:45.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.407+0000 7fa028833640 1 -- 172.21.15.99:0/3870280158 shutdown_connections 2024-03-25T18:27:45.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.407+0000 7fa028833640 1 -- 172.21.15.99:0/3870280158 wait complete. 2024-03-25T18:27:45.410 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.408+0000 7fa028833640 10 monclient: build_initial_monmap 2024-03-25T18:27:45.410 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.408+0000 7fa028833640 10 monclient: monmap: 2024-03-25T18:27:45.410 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:27:45.410 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:45.410 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:27:45.410 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:27:45.410 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:45.410 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:27:45.410 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:27:45.410 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:27:45.410 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:27:45.411 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:45.411 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.409+0000 7fa028833640 1 librados: starting msgr at 2024-03-25T18:27:45.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.409+0000 7fa028833640 1 librados: starting objecter 2024-03-25T18:27:45.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.409+0000 7fa028833640 1 Processor -- start 2024-03-25T18:27:45.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.409+0000 7fa028833640 1 -- start start 2024-03-25T18:27:45.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.409+0000 7fa028833640 1 librados: setting wanted keys 2024-03-25T18:27:45.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.409+0000 7fa028833640 1 librados: calling monclient init 2024-03-25T18:27:45.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.409+0000 7fa028833640 10 monclient: init 2024-03-25T18:27:45.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.410+0000 7fa028833640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:45.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.410+0000 7fa028833640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:45.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.410+0000 7fa028833640 10 monclient(hunting): picked mon.noname-c con 0x7fa004009af0 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:45.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.410+0000 7fa028833640 10 monclient(hunting): picked mon.noname-a con 0x7fa00400f530 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:45.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.410+0000 7fa028833640 10 monclient(hunting): picked mon.noname-b con 0x7fa004010730 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:45.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.410+0000 7fa028833640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa024140410 con 0x7fa00400f530 2024-03-25T18:27:45.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.410+0000 7fa028833640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0241dee70 con 0x7fa004010730 2024-03-25T18:27:45.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.410+0000 7fa028833640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02406ead0 con 0x7fa004009af0 2024-03-25T18:27:45.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.410+0000 7fa028833640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:45.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.410+0000 7fa028833640 10 monclient(hunting): authenticate will time out at 7483.578339s 2024-03-25T18:27:45.419 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.419+0000 7fa022d9d640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fa004010730 0x7fa0241de760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:38614/0 (socket says 172.21.15.99:38614) 2024-03-25T18:27:45.420 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.419+0000 7fa022d9d640 1 -- 172.21.15.99:0/3859640672 learned_addr learned my addr 172.21.15.99:0/3859640672 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:27:45.420 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.419+0000 7fa01affd640 10 client.?.objecter ms_handle_connect 0x7fa004010730 2024-03-25T18:27:45.420 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.419+0000 7fa01affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:45.423 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.423+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260236821 0 0) 0x7fa0241dee70 con 0x7fa004010730 2024-03-25T18:27:45.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.423+0000 7fa01affd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:45.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.423+0000 7fa01affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:45.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.423+0000 7fa01affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:45.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.423+0000 7fa01affd640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:27:45.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.423+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0040097a0 con 0x7fa004010730 2024-03-25T18:27:45.439 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.438+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2854176409 0 0) 0x7fa0040097a0 con 0x7fa004010730 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.438+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0241dee70 con 0x7fa004010730 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.438+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa01c001a80 con 0x7fa004010730 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.438+0000 7fa01affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.438+0000 7fa01affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.438+0000 7fa01affd640 10 monclient(hunting): dump: 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:45.440 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:45.441 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.438+0000 7fa01affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:45.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3238104832 0 0) 0x7fa0241dee70 con 0x7fa004010730 2024-03-25T18:27:45.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 >> v1:172.21.15.99:6790/0 conn(0x7fa004009af0 legacy=0x7fa0241408a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:45.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 >> v1:172.21.15.99:6789/0 conn(0x7fa00400f530 legacy=0x7fa0241de050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:45.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa01affd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:45.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa01affd640 1 monclient: found mon.b 2024-03-25T18:27:45.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa01affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:45.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa01affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:27:45.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02406ec60 con 0x7fa004010730 2024-03-25T18:27:45.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa01affd640 10 monclient: _finish_auth 0 2024-03-25T18:27:45.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa01affd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:45.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa01affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa028833640 5 monclient: authenticate success, global_id 4330 2024-03-25T18:27:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa028833640 10 monclient: _renew_subs 2024-03-25T18:27:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa028833640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:27:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa028833640 1 -- 172.21.15.99:0/3859640672 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa024141840 con 0x7fa004010730 2024-03-25T18:27:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa028833640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa028833640 10 monclient: _renew_subs 2024-03-25T18:27:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa028833640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:27:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa028833640 1 -- 172.21.15.99:0/3859640672 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa024136e60 con 0x7fa004010730 2024-03-25T18:27:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.453+0000 7fa028833640 1 librados: init done 2024-03-25T18:27:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.454+0000 7fa028833640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.454+0000 7fa028833640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.454+0000 7fa028833640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:27:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.454+0000 7fa028833640 1 -- 172.21.15.99:0/3859640672 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0240627b0 con 0x7fa004010730 2024-03-25T18:27:45.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.710+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa01c002030 con 0x7fa004010730 2024-03-25T18:27:45.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.710+0000 7fa01affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:45.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.710+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa01c003b10 con 0x7fa004010730 2024-03-25T18:27:45.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.710+0000 7fa01affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:45.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.710+0000 7fa01affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:45.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.710+0000 7fa01affd640 10 monclient: dump: 2024-03-25T18:27:45.711 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:45.711 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:45.712 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:45.712 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:45.712 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:45.712 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:45.712 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:45.712 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:45.712 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:45.712 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:45.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.710+0000 7fa020d99640 4 set_mon_vals no callback set 2024-03-25T18:27:45.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.711+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fa01c003d30 con 0x7fa004010730 2024-03-25T18:27:45.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.711+0000 7fa02259c640 10 monclient: get_auth_request con 0x7fa00406fd30 auth_method 0 2024-03-25T18:27:45.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.712+0000 7fa01affd640 10 client.4330.objecter ms_handle_connect 0x7fa00406fd30 2024-03-25T18:27:45.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.716+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fa01c0548f0 con 0x7fa004010730 2024-03-25T18:27:45.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.716+0000 7fa01affd640 10 client.4330.objecter ms_dispatch 0x7fa024000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:45.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.716+0000 7fa01affd640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:45.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.716+0000 7fa01affd640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:45.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.716+0000 7fa01affd640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-25T18:27:45.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.717+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa01c050400 con 0x7fa004010730 2024-03-25T18:27:45.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.717+0000 7fa01affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:45.719 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.717+0000 7fa01affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:45.938 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.937+0000 7fa028833640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:45.938 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.937+0000 7fa028833640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:45.939 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.937+0000 7fa028833640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:27:45.939 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:45.937+0000 7fa028833640 1 -- 172.21.15.99:0/3859640672 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa024069f40 con 0x7fa004010730 2024-03-25T18:27:46.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.110+0000 7fa01affd640 1 -- 172.21.15.99:0/3859640672 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fa01c052330 con 0x7fa004010730 2024-03-25T18:27:46.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.110+0000 7fa01affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:46.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.110+0000 7fa01affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:46.112 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:27:46.112 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:46.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.112+0000 7fa028833640 10 librados: watch_flush enter 2024-03-25T18:27:46.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.112+0000 7fa028833640 10 librados: watch_flush exit 2024-03-25T18:27:46.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.112+0000 7fa028833640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-25T18:27:46.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.112+0000 7fa028833640 10 client.4330.objecter successfully canceled tick 2024-03-25T18:27:46.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.112+0000 7fa028833640 1 -- 172.21.15.99:0/3859640672 >> v1:172.21.15.190:6800/4220409699 conn(0x7fa00406fd30 legacy=0x7fa0040721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:46.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.112+0000 7fa028833640 10 monclient: shutdown 2024-03-25T18:27:46.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.112+0000 7fa028833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:46.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.112+0000 7fa028833640 1 -- 172.21.15.99:0/3859640672 >> v1:172.21.15.190:6789/0 conn(0x7fa004010730 legacy=0x7fa0241de760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:46.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.112+0000 7fa028833640 1 -- 172.21.15.99:0/3859640672 shutdown_connections 2024-03-25T18:27:46.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.112+0000 7fa028833640 1 -- 172.21.15.99:0/3859640672 >> 172.21.15.99:0/3859640672 conn(0x7fa0240617a0 msgr2=0x7fa0240b58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:46.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.112+0000 7fa028833640 1 -- 172.21.15.99:0/3859640672 shutdown_connections 2024-03-25T18:27:46.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.112+0000 7fa028833640 1 -- 172.21.15.99:0/3859640672 wait complete. 2024-03-25T18:27:46.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:46.112+0000 7fa028833640 1 librados: shutdown 2024-03-25T18:27:46.124 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:48.605 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:02:49.197 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:48.616 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:02:49.208 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:48.651 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:02:49.244 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:48.658 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:02:49.216 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:48.668 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:02:49.227 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:48.680 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:02:49.273 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:48.683 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:02:49.276 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:48.710 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:02:49.269 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:48.720 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:02:49.313 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:52.932 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:53.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.046+0000 7fe2849c8640 10 monclient: get_monmap_and_config 2024-03-25T18:27:53.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.046+0000 7fe2849c8640 10 monclient: build_initial_monmap 2024-03-25T18:27:53.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.046+0000 7fe2849c8640 10 monclient: monmap: 2024-03-25T18:27:53.048 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:27:53.048 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:53.048 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:27:53.047802+0000 2024-03-25T18:27:53.048 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:27:53.047802+0000 2024-03-25T18:27:53.048 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:53.048 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:53.048 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:27:53.048 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:27:53.048 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:27:53.048 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:53.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.047+0000 7fe2849c8640 1 Processor -- start 2024-03-25T18:27:53.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.047+0000 7fe2849c8640 1 -- start start 2024-03-25T18:27:53.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.047+0000 7fe2849c8640 10 monclient: init 2024-03-25T18:27:53.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.048+0000 7fe2849c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:53.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.048+0000 7fe2849c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:53.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.048+0000 7fe2849c8640 10 monclient(hunting): picked mon.noname-b con 0x7fe2800b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:53.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.048+0000 7fe2849c8640 10 monclient(hunting): picked mon.noname-c con 0x7fe2800b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:53.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.048+0000 7fe2849c8640 10 monclient(hunting): picked mon.noname-a con 0x7fe28014be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:53.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.048+0000 7fe2849c8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2800af940 con 0x7fe28014be90 2024-03-25T18:27:53.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.048+0000 7fe2849c8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2800b0a70 con 0x7fe2800b1730 2024-03-25T18:27:53.050 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.048+0000 7fe2849c8640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2800b0d50 con 0x7fe2800b4d60 2024-03-25T18:27:53.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.048+0000 7fe2849c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:53.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.048+0000 7fe2849c8640 10 monclient(hunting): authenticate will time out at 7491.216559s 2024-03-25T18:27:53.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.056+0000 7fe277fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe2800b1730 0x7fe2800b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:42782/0 (socket says 172.21.15.99:42782) 2024-03-25T18:27:53.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.056+0000 7fe277fff640 1 -- 172.21.15.99:0/1825380203 learned_addr learned my addr 172.21.15.99:0/1825380203 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:27:53.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.074+0000 7fe27dd9b640 1 -- 172.21.15.99:0/1825380203 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186319976 0 0) 0x7fe2800af940 con 0x7fe28014be90 2024-03-25T18:27:53.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.075+0000 7fe27dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:27:53.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.075+0000 7fe27dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:53.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.075+0000 7fe27dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:53.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.075+0000 7fe27dd9b640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:27:53.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.075+0000 7fe27dd9b640 1 -- 172.21.15.99:0/1825380203 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe26c003520 con 0x7fe28014be90 2024-03-25T18:27:53.127 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.126+0000 7fe27dd9b640 1 -- 172.21.15.99:0/1825380203 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154168276 0 0) 0x7fe2800b0a70 con 0x7fe2800b1730 2024-03-25T18:27:53.127 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.126+0000 7fe27dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:27:53.127 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.126+0000 7fe27dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:53.127 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.126+0000 7fe27dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:53.127 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.126+0000 7fe27dd9b640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:27:53.127 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.126+0000 7fe27dd9b640 1 -- 172.21.15.99:0/1825380203 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2800af940 con 0x7fe2800b1730 2024-03-25T18:27:53.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 1 -- 172.21.15.99:0/1825380203 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3936851855 0 0) 0x7fe26c003520 con 0x7fe28014be90 2024-03-25T18:27:53.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 1 -- 172.21.15.99:0/1825380203 >> v1:172.21.15.99:6790/0 conn(0x7fe2800b4d60 legacy=0x7fe28013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:53.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 1 -- 172.21.15.99:0/1825380203 >> v1:172.21.15.190:6789/0 conn(0x7fe2800b1730 legacy=0x7fe2800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:53.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:27:53.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 1 monclient: found mon.noname-a 2024-03-25T18:27:53.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:53.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:27:53.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 1 -- 172.21.15.99:0/1825380203 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2800b1030 con 0x7fe28014be90 2024-03-25T18:27:53.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:27:53.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:27:53.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:53.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 1 -- 172.21.15.99:0/1825380203 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe278001a10 con 0x7fe28014be90 2024-03-25T18:27:53.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe2849c8640 5 monclient: authenticate success, global_id 4317 2024-03-25T18:27:53.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe2849c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:53.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:53.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:53.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe27dd9b640 10 monclient: dump: 2024-03-25T18:27:53.135 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:53.135 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:53.135 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:53.136 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:53.136 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:53.136 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:53.136 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:53.136 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:53.136 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:53.136 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:53.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.132+0000 7fe2849c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:53.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.159+0000 7fe27dd9b640 1 -- 172.21.15.99:0/1825380203 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe278001b70 con 0x7fe28014be90 2024-03-25T18:27:53.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.159+0000 7fe27dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:53.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.159+0000 7fe27dd9b640 1 -- 172.21.15.99:0/1825380203 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe278001f80 con 0x7fe28014be90 2024-03-25T18:27:53.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.159+0000 7fe27dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:53.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.159+0000 7fe27dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:53.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.159+0000 7fe27dd9b640 10 monclient: dump: 2024-03-25T18:27:53.161 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:53.161 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:53.161 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:53.161 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:53.161 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:53.161 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:53.161 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:53.161 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:53.161 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:53.161 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:53.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.159+0000 7fe2849c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:53.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.159+0000 7fe2849c8640 4 set_mon_vals no callback set 2024-03-25T18:27:53.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.160+0000 7fe2849c8640 10 monclient: shutdown 2024-03-25T18:27:53.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.160+0000 7fe2849c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:53.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.160+0000 7fe2849c8640 1 -- 172.21.15.99:0/1825380203 >> v1:172.21.15.99:6789/0 conn(0x7fe28014be90 legacy=0x7fe28014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:53.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.160+0000 7fe2849c8640 1 -- 172.21.15.99:0/1825380203 shutdown_connections 2024-03-25T18:27:53.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.160+0000 7fe2849c8640 1 -- 172.21.15.99:0/1825380203 >> 172.21.15.99:0/1825380203 conn(0x7fe2800617a0 msgr2=0x7fe280061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:53.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.160+0000 7fe2849c8640 1 -- 172.21.15.99:0/1825380203 shutdown_connections 2024-03-25T18:27:53.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.160+0000 7fe2849c8640 1 -- 172.21.15.99:0/1825380203 wait complete. 2024-03-25T18:27:53.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.161+0000 7fe2849c8640 10 monclient: build_initial_monmap 2024-03-25T18:27:53.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.161+0000 7fe2849c8640 10 monclient: monmap: 2024-03-25T18:27:53.165 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:27:53.165 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:53.165 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:27:53.165 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:27:53.165 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:53.165 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:27:53.165 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:27:53.166 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:27:53.166 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:27:53.166 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:53.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.161+0000 7fe2849c8640 1 librados: starting msgr at 2024-03-25T18:27:53.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.161+0000 7fe2849c8640 1 librados: starting objecter 2024-03-25T18:27:53.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.162+0000 7fe2849c8640 1 Processor -- start 2024-03-25T18:27:53.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.162+0000 7fe2849c8640 1 -- start start 2024-03-25T18:27:53.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.162+0000 7fe2849c8640 1 librados: setting wanted keys 2024-03-25T18:27:53.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.162+0000 7fe2849c8640 1 librados: calling monclient init 2024-03-25T18:27:53.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.162+0000 7fe2849c8640 10 monclient: init 2024-03-25T18:27:53.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.162+0000 7fe2849c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:53.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.162+0000 7fe2849c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:53.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.162+0000 7fe2849c8640 10 monclient(hunting): picked mon.noname-b con 0x7fe2800b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:27:53.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.163+0000 7fe2849c8640 10 monclient(hunting): picked mon.noname-c con 0x7fe2800b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:27:53.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.163+0000 7fe2849c8640 10 monclient(hunting): picked mon.noname-a con 0x7fe28014be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:27:53.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.163+0000 7fe2849c8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe280141950 con 0x7fe28014be90 2024-03-25T18:27:53.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.163+0000 7fe2849c8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2801d7940 con 0x7fe2800b1730 2024-03-25T18:27:53.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.163+0000 7fe2849c8640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2801d8570 con 0x7fe2800b4d60 2024-03-25T18:27:53.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.163+0000 7fe2849c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:53.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.163+0000 7fe2849c8640 10 monclient(hunting): authenticate will time out at 7491.331284s 2024-03-25T18:27:53.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.169+0000 7fe27e59c640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe2800b1730 0x7fe2801d7e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:42826/0 (socket says 172.21.15.99:42826) 2024-03-25T18:27:53.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.169+0000 7fe27e59c640 1 -- 172.21.15.99:0/1799109512 learned_addr learned my addr 172.21.15.99:0/1799109512 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:27:53.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.171+0000 7fe276ffd640 10 client.?.objecter ms_handle_connect 0x7fe2800b1730 2024-03-25T18:27:53.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.171+0000 7fe276ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:53.174 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.173+0000 7fe276ffd640 10 client.?.objecter ms_handle_connect 0x7fe2800b4d60 2024-03-25T18:27:53.175 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.173+0000 7fe276ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:53.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.176+0000 7fe276ffd640 10 client.?.objecter ms_handle_connect 0x7fe28014be90 2024-03-25T18:27:53.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.176+0000 7fe276ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:53.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.179+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402738088 0 0) 0x7fe2801d8570 con 0x7fe2800b4d60 2024-03-25T18:27:53.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.179+0000 7fe276ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:53.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.179+0000 7fe276ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:53.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.179+0000 7fe276ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:53.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.179+0000 7fe276ffd640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:27:53.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.179+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe260004eb0 con 0x7fe2800b4d60 2024-03-25T18:27:53.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.183+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635874533 0 0) 0x7fe280141950 con 0x7fe28014be90 2024-03-25T18:27:53.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.183+0000 7fe276ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:53.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.183+0000 7fe276ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:53.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.183+0000 7fe276ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:53.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.183+0000 7fe276ffd640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:27:53.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.183+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2801d8570 con 0x7fe28014be90 2024-03-25T18:27:53.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.188+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614631893 0 0) 0x7fe2801d7940 con 0x7fe2800b1730 2024-03-25T18:27:53.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.188+0000 7fe276ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:53.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.188+0000 7fe276ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:53.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.188+0000 7fe276ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:53.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.188+0000 7fe276ffd640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:27:53.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.188+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe280141950 con 0x7fe2800b1730 2024-03-25T18:27:53.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.197+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2872580675 0 0) 0x7fe2801d8570 con 0x7fe28014be90 2024-03-25T18:27:53.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.197+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2801d7940 con 0x7fe28014be90 2024-03-25T18:27:53.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.197+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe278001e80 con 0x7fe28014be90 2024-03-25T18:27:53.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.197+0000 7fe276ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:53.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.197+0000 7fe276ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:53.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.197+0000 7fe276ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:53.199 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:53.199 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:53.199 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:53.199 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:53.199 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:53.199 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:53.199 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:53.199 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:53.199 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:53.199 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:53.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.197+0000 7fe276ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.206+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2304156490 0 0) 0x7fe280141950 con 0x7fe2800b1730 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.206+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2801d8570 con 0x7fe2800b1730 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.206+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe2700018f0 con 0x7fe2800b1730 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.206+0000 7fe276ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.206+0000 7fe276ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.206+0000 7fe276ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:53.207 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:53.208 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:53.217 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.216+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2203021361 0 0) 0x7fe260004eb0 con 0x7fe2800b4d60 2024-03-25T18:27:53.217 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.216+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe280141950 con 0x7fe2800b4d60 2024-03-25T18:27:53.217 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.216+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe268001930 con 0x7fe2800b4d60 2024-03-25T18:27:53.217 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.216+0000 7fe276ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:53.217 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.216+0000 7fe276ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:53.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.216+0000 7fe276ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:53.218 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:53.218 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:53.218 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:53.218 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:53.218 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:53.218 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:53.218 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:53.218 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:53.218 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:53.218 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:53.220 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2716810228 0 0) 0x7fe2801d8570 con 0x7fe2800b1730 2024-03-25T18:27:53.220 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 >> v1:172.21.15.99:6790/0 conn(0x7fe2800b4d60 legacy=0x7fe280142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:53.220 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 >> v1:172.21.15.99:6789/0 conn(0x7fe28014be90 legacy=0x7fe2801e2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:53.221 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe276ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:53.221 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe276ffd640 1 monclient: found mon.b 2024-03-25T18:27:53.221 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe276ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:53.221 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe276ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:27:53.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe280138840 con 0x7fe2800b1730 2024-03-25T18:27:53.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe276ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:53.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe276ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:53.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe276ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:53.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe2849c8640 5 monclient: authenticate success, global_id 4336 2024-03-25T18:27:53.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe2849c8640 10 monclient: _renew_subs 2024-03-25T18:27:53.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe2849c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:27:53.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe2849c8640 1 -- 172.21.15.99:0/1799109512 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe280142f70 con 0x7fe2800b1730 2024-03-25T18:27:53.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe2849c8640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:53.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe2849c8640 10 monclient: _renew_subs 2024-03-25T18:27:53.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe2849c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:27:53.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe2849c8640 1 -- 172.21.15.99:0/1799109512 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe280137da0 con 0x7fe2800b1730 2024-03-25T18:27:53.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.219+0000 7fe2849c8640 1 librados: init done 2024-03-25T18:27:53.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.220+0000 7fe2849c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:53.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.220+0000 7fe2849c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:53.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.220+0000 7fe2849c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:27:53.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.220+0000 7fe2849c8640 1 -- 172.21.15.99:0/1799109512 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2801d7ad0 con 0x7fe2800b1730 2024-03-25T18:27:53.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.494+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe270003610 con 0x7fe2800b1730 2024-03-25T18:27:53.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.494+0000 7fe276ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:53.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.494+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe2700039e0 con 0x7fe2800b1730 2024-03-25T18:27:53.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.494+0000 7fe276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:53.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.494+0000 7fe276ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:53.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.494+0000 7fe276ffd640 10 monclient: dump: 2024-03-25T18:27:53.496 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:27:53.496 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:27:53.496 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:53.496 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:27:53.496 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:53.496 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:27:53.496 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:27:53.496 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:27:53.496 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:27:53.496 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:27:53.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.494+0000 7fe27d59a640 4 set_mon_vals no callback set 2024-03-25T18:27:53.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.495+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fe270003c00 con 0x7fe2800b1730 2024-03-25T18:27:53.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.496+0000 7fe277fff640 10 monclient: get_auth_request con 0x7fe26006e160 auth_method 0 2024-03-25T18:27:53.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.496+0000 7fe276ffd640 10 client.4336.objecter ms_handle_connect 0x7fe26006e160 2024-03-25T18:27:53.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.500+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fe270054730 con 0x7fe2800b1730 2024-03-25T18:27:53.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.500+0000 7fe276ffd640 10 client.4336.objecter ms_dispatch 0x7fe280000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:53.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.500+0000 7fe276ffd640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:53.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.500+0000 7fe276ffd640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:53.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.500+0000 7fe276ffd640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T18:27:53.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.500+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe270050240 con 0x7fe2800b1730 2024-03-25T18:27:53.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.500+0000 7fe276ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:53.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.500+0000 7fe276ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:53.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.745+0000 7fe2849c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:53.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.745+0000 7fe2849c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:53.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.745+0000 7fe2849c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:27:53.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.745+0000 7fe2849c8640 1 -- 172.21.15.99:0/1799109512 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2801389d0 con 0x7fe2800b1730 2024-03-25T18:27:53.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.767+0000 7fe276ffd640 1 -- 172.21.15.99:0/1799109512 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fe270052170 con 0x7fe2800b1730 2024-03-25T18:27:53.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.767+0000 7fe276ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:53.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.767+0000 7fe276ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:53.769 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:27:53.769 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:53.771 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.769+0000 7fe2849c8640 10 librados: watch_flush enter 2024-03-25T18:27:53.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.770+0000 7fe2849c8640 10 librados: watch_flush exit 2024-03-25T18:27:53.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.770+0000 7fe2849c8640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T18:27:53.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.770+0000 7fe2849c8640 10 client.4336.objecter successfully canceled tick 2024-03-25T18:27:53.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.770+0000 7fe2849c8640 1 -- 172.21.15.99:0/1799109512 >> v1:172.21.15.190:6800/4220409699 conn(0x7fe26006e160 legacy=0x7fe260070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:53.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.770+0000 7fe2849c8640 10 monclient: shutdown 2024-03-25T18:27:53.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.770+0000 7fe2849c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:53.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.770+0000 7fe2849c8640 1 -- 172.21.15.99:0/1799109512 >> v1:172.21.15.190:6789/0 conn(0x7fe2800b1730 legacy=0x7fe2801d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:53.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.770+0000 7fe2849c8640 1 -- 172.21.15.99:0/1799109512 shutdown_connections 2024-03-25T18:27:53.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.770+0000 7fe2849c8640 1 -- 172.21.15.99:0/1799109512 >> 172.21.15.99:0/1799109512 conn(0x7fe2800617a0 msgr2=0x7fe2800b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:53.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.770+0000 7fe2849c8640 1 -- 172.21.15.99:0/1799109512 shutdown_connections 2024-03-25T18:27:53.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.770+0000 7fe2849c8640 1 -- 172.21.15.99:0/1799109512 wait complete. 2024-03-25T18:27:53.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:27:53.770+0000 7fe2849c8640 1 librados: shutdown 2024-03-25T18:27:53.783 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:00.591 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:00.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.706+0000 7f2eea3f4640 10 monclient: get_monmap_and_config 2024-03-25T18:28:00.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.706+0000 7f2eea3f4640 10 monclient: build_initial_monmap 2024-03-25T18:28:00.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.706+0000 7f2eea3f4640 10 monclient: monmap: 2024-03-25T18:28:00.708 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:00.708 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:00.708 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:28:00.707880+0000 2024-03-25T18:28:00.708 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:28:00.707880+0000 2024-03-25T18:28:00.708 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:00.708 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:00.708 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:00.708 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:28:00.708 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:28:00.708 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:00.710 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.707+0000 7f2eea3f4640 1 Processor -- start 2024-03-25T18:28:00.710 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.707+0000 7f2eea3f4640 1 -- start start 2024-03-25T18:28:00.710 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.707+0000 7f2eea3f4640 10 monclient: init 2024-03-25T18:28:00.710 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.708+0000 7f2eea3f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:00.710 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.708+0000 7f2eea3f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:00.710 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.708+0000 7f2eea3f4640 10 monclient(hunting): picked mon.noname-a con 0x7f2ee40a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:00.710 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.708+0000 7f2eea3f4640 10 monclient(hunting): picked mon.noname-c con 0x7f2ee413f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:00.710 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.708+0000 7f2eea3f4640 10 monclient(hunting): picked mon.noname-b con 0x7f2ee41430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:00.710 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.708+0000 7f2eea3f4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee4142520 con 0x7f2ee40a98f0 2024-03-25T18:28:00.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.708+0000 7f2eea3f4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee40b28a0 con 0x7f2ee413f710 2024-03-25T18:28:00.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.708+0000 7f2eea3f4640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee40b2a30 con 0x7f2ee41430c0 2024-03-25T18:28:00.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.708+0000 7f2eea3f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:00.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.708+0000 7f2eea3f4640 10 monclient(hunting): authenticate will time out at 7498.876407s 2024-03-25T18:28:00.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.714+0000 7f2edbfff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f2ee413f710 0x7f2ee41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:43270/0 (socket says 172.21.15.99:43270) 2024-03-25T18:28:00.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.714+0000 7f2edbfff640 1 -- 172.21.15.99:0/2499749878 learned_addr learned my addr 172.21.15.99:0/2499749878 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:00.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.727+0000 7f2ee37fe640 1 -- 172.21.15.99:0/2499749878 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349234997 0 0) 0x7f2ee40b28a0 con 0x7f2ee413f710 2024-03-25T18:28:00.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.727+0000 7f2ee37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:00.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.727+0000 7f2ee37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:00.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.727+0000 7f2ee37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:00.729 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.727+0000 7f2ee37fe640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:28:00.729 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.727+0000 7f2ee37fe640 1 -- 172.21.15.99:0/2499749878 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed0003520 con 0x7f2ee413f710 2024-03-25T18:28:00.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.734+0000 7f2ee37fe640 1 -- 172.21.15.99:0/2499749878 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846333293 0 0) 0x7f2ee4142520 con 0x7f2ee40a98f0 2024-03-25T18:28:00.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.734+0000 7f2ee37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:00.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.734+0000 7f2ee37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:00.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.734+0000 7f2ee37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:00.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.734+0000 7f2ee37fe640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:28:00.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.734+0000 7f2ee37fe640 1 -- 172.21.15.99:0/2499749878 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee40b28a0 con 0x7f2ee40a98f0 2024-03-25T18:28:00.742 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.740+0000 7f2ee37fe640 1 -- 172.21.15.99:0/2499749878 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2386146533 0 0) 0x7f2ed0003520 con 0x7f2ee413f710 2024-03-25T18:28:00.742 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.740+0000 7f2ee37fe640 1 -- 172.21.15.99:0/2499749878 >> v1:172.21.15.99:6790/0 conn(0x7f2ee41430c0 legacy=0x7f2ee414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:00.742 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.740+0000 7f2ee37fe640 1 -- 172.21.15.99:0/2499749878 >> v1:172.21.15.99:6789/0 conn(0x7f2ee40a98f0 legacy=0x7f2ee40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:00.742 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.740+0000 7f2ee37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:00.742 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.740+0000 7f2ee37fe640 1 monclient: found mon.noname-c 2024-03-25T18:28:00.742 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.741+0000 7f2ee37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:00.742 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.741+0000 7f2ee37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:28:00.742 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.741+0000 7f2ee37fe640 1 -- 172.21.15.99:0/2499749878 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee40b44c0 con 0x7f2ee413f710 2024-03-25T18:28:00.742 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.741+0000 7f2ee37fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:00.742 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.741+0000 7f2ee37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:00.743 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.741+0000 7f2ee37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:00.743 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.741+0000 7f2ee37fe640 1 -- 172.21.15.99:0/2499749878 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2ecc001690 con 0x7f2ee413f710 2024-03-25T18:28:00.743 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.741+0000 7f2ee37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:00.743 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.741+0000 7f2ee37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:00.743 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.741+0000 7f2ee37fe640 10 monclient: dump: 2024-03-25T18:28:00.743 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:00.743 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:00.743 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:00.743 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:00.743 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:00.743 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:00.743 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:00.743 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:00.743 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:00.744 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:00.744 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.741+0000 7f2eea3f4640 5 monclient: authenticate success, global_id 4339 2024-03-25T18:28:00.744 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.741+0000 7f2eea3f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:00.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.760+0000 7f2ee37fe640 1 -- 172.21.15.99:0/2499749878 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ecc0017f0 con 0x7f2ee413f710 2024-03-25T18:28:00.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.760+0000 7f2ee37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:00.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.760+0000 7f2ee37fe640 1 -- 172.21.15.99:0/2499749878 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2ecc001c00 con 0x7f2ee413f710 2024-03-25T18:28:00.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.760+0000 7f2eea3f4640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:00.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.760+0000 7f2eea3f4640 4 set_mon_vals no callback set 2024-03-25T18:28:00.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.760+0000 7f2ee37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:00.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.760+0000 7f2ee37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:00.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.760+0000 7f2ee37fe640 10 monclient: dump: 2024-03-25T18:28:00.762 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:00.762 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:00.762 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:00.762 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:00.762 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:00.762 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:00.762 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:00.762 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:00.762 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:00.762 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:00.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.761+0000 7f2eea3f4640 10 monclient: shutdown 2024-03-25T18:28:00.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.761+0000 7f2eea3f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:00.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.761+0000 7f2eea3f4640 1 -- 172.21.15.99:0/2499749878 >> v1:172.21.15.190:6789/0 conn(0x7f2ee413f710 legacy=0x7f2ee41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:00.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.761+0000 7f2eea3f4640 1 -- 172.21.15.99:0/2499749878 shutdown_connections 2024-03-25T18:28:00.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.761+0000 7f2eea3f4640 1 -- 172.21.15.99:0/2499749878 >> 172.21.15.99:0/2499749878 conn(0x7f2ee40b7ea0 msgr2=0x7f2ee40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:00.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.761+0000 7f2eea3f4640 1 -- 172.21.15.99:0/2499749878 shutdown_connections 2024-03-25T18:28:00.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.761+0000 7f2eea3f4640 1 -- 172.21.15.99:0/2499749878 wait complete. 2024-03-25T18:28:00.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.762+0000 7f2eea3f4640 10 monclient: build_initial_monmap 2024-03-25T18:28:00.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.762+0000 7f2eea3f4640 10 monclient: monmap: 2024-03-25T18:28:00.765 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:00.765 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:00.765 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:28:00.765 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:28:00.766 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:00.766 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:28:00.766 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:00.766 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:28:00.766 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:28:00.766 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:00.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.762+0000 7f2eea3f4640 1 librados: starting msgr at 2024-03-25T18:28:00.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.762+0000 7f2eea3f4640 1 librados: starting objecter 2024-03-25T18:28:00.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.762+0000 7f2eea3f4640 1 Processor -- start 2024-03-25T18:28:00.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.762+0000 7f2eea3f4640 1 -- start start 2024-03-25T18:28:00.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.762+0000 7f2eea3f4640 1 librados: setting wanted keys 2024-03-25T18:28:00.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.762+0000 7f2eea3f4640 1 librados: calling monclient init 2024-03-25T18:28:00.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.762+0000 7f2eea3f4640 10 monclient: init 2024-03-25T18:28:00.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.763+0000 7f2eea3f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:00.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.763+0000 7f2eea3f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:00.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.763+0000 7f2eea3f4640 10 monclient(hunting): picked mon.noname-c con 0x7f2ee40a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:00.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.763+0000 7f2eea3f4640 10 monclient(hunting): picked mon.noname-a con 0x7f2ee413f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:00.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.763+0000 7f2eea3f4640 10 monclient(hunting): picked mon.noname-b con 0x7f2ee41430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:00.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.763+0000 7f2eea3f4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee4138a50 con 0x7f2ee413f710 2024-03-25T18:28:00.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.763+0000 7f2eea3f4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee40ab9e0 con 0x7f2ee41430c0 2024-03-25T18:28:00.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.763+0000 7f2eea3f4640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee41d7880 con 0x7f2ee40a98f0 2024-03-25T18:28:00.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.763+0000 7f2eea3f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:00.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.763+0000 7f2eea3f4640 10 monclient(hunting): authenticate will time out at 7498.931590s 2024-03-25T18:28:00.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.779+0000 7f2edbfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2ee413f710 0x7f2ee41d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56730/0 (socket says 172.21.15.99:56730) 2024-03-25T18:28:00.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.779+0000 7f2edbfff640 1 -- 172.21.15.99:0/4282289381 learned_addr learned my addr 172.21.15.99:0/4282289381 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:00.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.782+0000 7f2ee17fa640 10 client.?.objecter ms_handle_connect 0x7f2ee413f710 2024-03-25T18:28:00.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.782+0000 7f2ee17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:00.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.788+0000 7f2ee17fa640 10 client.?.objecter ms_handle_connect 0x7f2ee41430c0 2024-03-25T18:28:00.789 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.788+0000 7f2ee17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:00.790 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.789+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158623670 0 0) 0x7f2ee4138a50 con 0x7f2ee413f710 2024-03-25T18:28:00.790 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.789+0000 7f2ee17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:00.790 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.789+0000 7f2ee17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:00.790 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.789+0000 7f2ee17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:00.790 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.789+0000 7f2ee17fa640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:28:00.791 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.789+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec4004eb0 con 0x7f2ee413f710 2024-03-25T18:28:00.795 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.794+0000 7f2ee17fa640 10 client.?.objecter ms_handle_connect 0x7f2ee40a98f0 2024-03-25T18:28:00.795 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.794+0000 7f2ee17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:00.801 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.800+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925532221 0 0) 0x7f2ee40ab9e0 con 0x7f2ee41430c0 2024-03-25T18:28:00.801 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.800+0000 7f2ee17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:00.801 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.800+0000 7f2ee17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:00.801 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.800+0000 7f2ee17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:00.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.800+0000 7f2ee17fa640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:28:00.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.800+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee4138a50 con 0x7f2ee41430c0 2024-03-25T18:28:00.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.800+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716456292 0 0) 0x7f2ee41d7880 con 0x7f2ee40a98f0 2024-03-25T18:28:00.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.800+0000 7f2ee17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:00.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.800+0000 7f2ee17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:00.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.800+0000 7f2ee17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:00.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.800+0000 7f2ee17fa640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:28:00.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.800+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee40ab9e0 con 0x7f2ee40a98f0 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.804+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2915192864 0 0) 0x7f2ec4004eb0 con 0x7f2ee413f710 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.804+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ee41d7880 con 0x7f2ee413f710 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.804+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2ecc001bc0 con 0x7f2ee413f710 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.804+0000 7f2ee17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.804+0000 7f2ee17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.804+0000 7f2ee17fa640 10 monclient(hunting): dump: 2024-03-25T18:28:00.805 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:00.806 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:00.806 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:00.806 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:00.806 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:00.806 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:00.806 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:00.806 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:00.806 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:00.806 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:00.806 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.804+0000 7f2ee17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:00.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.807+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3968076263 0 0) 0x7f2ee4138a50 con 0x7f2ee41430c0 2024-03-25T18:28:00.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.807+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ec4004eb0 con 0x7f2ee41430c0 2024-03-25T18:28:00.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.807+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2edc001b10 con 0x7f2ee41430c0 2024-03-25T18:28:00.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.807+0000 7f2ee17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:00.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.807+0000 7f2ee17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:00.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.807+0000 7f2ee17fa640 10 monclient(hunting): dump: 2024-03-25T18:28:00.809 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:00.809 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:00.809 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:00.809 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:00.809 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:00.809 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:00.810 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:00.810 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:00.810 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:00.810 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:00.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2032337206 0 0) 0x7f2ec4004eb0 con 0x7f2ee41430c0 2024-03-25T18:28:00.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 >> v1:172.21.15.99:6790/0 conn(0x7f2ee40a98f0 legacy=0x7f2ee4138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:00.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 >> v1:172.21.15.99:6789/0 conn(0x7f2ee413f710 legacy=0x7f2ee41d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:00.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2ee17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:28:00.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2ee17fa640 1 monclient: found mon.b 2024-03-25T18:28:00.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2ee17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:00.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2ee17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:00.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee41d84b0 con 0x7f2ee41430c0 2024-03-25T18:28:00.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2ee17fa640 10 monclient: _finish_auth 0 2024-03-25T18:28:00.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2ee17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:28:00.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2ee17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:00.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2eea3f4640 5 monclient: authenticate success, global_id 4342 2024-03-25T18:28:00.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2eea3f4640 10 monclient: _renew_subs 2024-03-25T18:28:00.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2eea3f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:00.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2eea3f4640 1 -- 172.21.15.99:0/4282289381 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee41d51f0 con 0x7f2ee41430c0 2024-03-25T18:28:00.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2eea3f4640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:00.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2eea3f4640 10 monclient: _renew_subs 2024-03-25T18:28:00.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2eea3f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:00.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2eea3f4640 1 -- 172.21.15.99:0/4282289381 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee4138e60 con 0x7f2ee41430c0 2024-03-25T18:28:00.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.817+0000 7f2eea3f4640 1 librados: init done 2024-03-25T18:28:00.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.818+0000 7f2eea3f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:00.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.818+0000 7f2eea3f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:00.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.818+0000 7f2eea3f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:00.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:00.818+0000 7f2eea3f4640 1 -- 172.21.15.99:0/4282289381 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee41d76e0 con 0x7f2ee41430c0 2024-03-25T18:28:01.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.082+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2edc002080 con 0x7f2ee41430c0 2024-03-25T18:28:01.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.082+0000 7f2ee17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:01.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.082+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2edc003990 con 0x7f2ee41430c0 2024-03-25T18:28:01.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.082+0000 7f2ee17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:01.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.082+0000 7f2ee17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:01.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.082+0000 7f2ee2ffd640 4 set_mon_vals no callback set 2024-03-25T18:28:01.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.082+0000 7f2ee17fa640 10 monclient: dump: 2024-03-25T18:28:01.083 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:01.083 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:01.084 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:01.084 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:01.084 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:01.084 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:01.084 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:01.084 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:01.084 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:01.084 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:01.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.083+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f2edc003b70 con 0x7f2ee41430c0 2024-03-25T18:28:01.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.083+0000 7f2ee3fff640 10 monclient: get_auth_request con 0x7f2ec406df60 auth_method 0 2024-03-25T18:28:01.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.084+0000 7f2ee17fa640 10 client.4342.objecter ms_handle_connect 0x7f2ec406df60 2024-03-25T18:28:01.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.087+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f2edc053730 con 0x7f2ee41430c0 2024-03-25T18:28:01.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.087+0000 7f2ee17fa640 10 client.4342.objecter ms_dispatch 0x7f2ee4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:01.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.087+0000 7f2ee17fa640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:01.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.087+0000 7f2ee17fa640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:01.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.087+0000 7f2ee17fa640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-25T18:28:01.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.087+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2edc04f070 con 0x7f2ee41430c0 2024-03-25T18:28:01.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.087+0000 7f2ee17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:01.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.087+0000 7f2ee17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:01.313 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.312+0000 7f2eea3f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:01.314 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.312+0000 7f2eea3f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:01.314 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.312+0000 7f2eea3f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:01.314 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.312+0000 7f2eea3f4640 1 -- 172.21.15.99:0/4282289381 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ee40627b0 con 0x7f2ee41430c0 2024-03-25T18:28:01.334 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.333+0000 7f2ee17fa640 1 -- 172.21.15.99:0/4282289381 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f2edc051170 con 0x7f2ee41430c0 2024-03-25T18:28:01.334 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.333+0000 7f2ee17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:01.334 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.333+0000 7f2ee17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:01.334 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:28:01.334 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:01.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.335+0000 7f2eea3f4640 10 librados: watch_flush enter 2024-03-25T18:28:01.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.335+0000 7f2eea3f4640 10 librados: watch_flush exit 2024-03-25T18:28:01.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.335+0000 7f2eea3f4640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-25T18:28:01.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.335+0000 7f2eea3f4640 10 client.4342.objecter successfully canceled tick 2024-03-25T18:28:01.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.335+0000 7f2eea3f4640 1 -- 172.21.15.99:0/4282289381 >> v1:172.21.15.190:6800/4220409699 conn(0x7f2ec406df60 legacy=0x7f2ec4070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:01.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.335+0000 7f2eea3f4640 10 monclient: shutdown 2024-03-25T18:28:01.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.335+0000 7f2eea3f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:01.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.335+0000 7f2eea3f4640 1 -- 172.21.15.99:0/4282289381 >> v1:172.21.15.190:6789/0 conn(0x7f2ee41430c0 legacy=0x7f2ee41e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:01.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.335+0000 7f2eea3f4640 1 -- 172.21.15.99:0/4282289381 shutdown_connections 2024-03-25T18:28:01.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.335+0000 7f2eea3f4640 1 -- 172.21.15.99:0/4282289381 >> 172.21.15.99:0/4282289381 conn(0x7f2ee40b7ea0 msgr2=0x7f2ee4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:01.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.335+0000 7f2eea3f4640 1 -- 172.21.15.99:0/4282289381 shutdown_connections 2024-03-25T18:28:01.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.335+0000 7f2eea3f4640 1 -- 172.21.15.99:0/4282289381 wait complete. 2024-03-25T18:28:01.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:01.335+0000 7f2eea3f4640 1 librados: shutdown 2024-03-25T18:28:01.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:08.154 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:08.274 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.273+0000 7f2a4c97b640 10 monclient: get_monmap_and_config 2024-03-25T18:28:08.274 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.273+0000 7f2a4c97b640 10 monclient: build_initial_monmap 2024-03-25T18:28:08.274 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.273+0000 7f2a4c97b640 10 monclient: monmap: 2024-03-25T18:28:08.275 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:08.275 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:08.275 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:28:08.274125+0000 2024-03-25T18:28:08.275 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:28:08.274125+0000 2024-03-25T18:28:08.275 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:08.275 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:08.275 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:08.275 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:28:08.275 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:28:08.275 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:08.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.273+0000 7f2a4c97b640 1 Processor -- start 2024-03-25T18:28:08.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.273+0000 7f2a4c97b640 1 -- start start 2024-03-25T18:28:08.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.273+0000 7f2a4c97b640 10 monclient: init 2024-03-25T18:28:08.276 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.274+0000 7f2a4c97b640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:08.276 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.274+0000 7f2a4c97b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:08.277 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.274+0000 7f2a4c97b640 10 monclient(hunting): picked mon.noname-a con 0x7f2a480b4d60 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:08.277 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.274+0000 7f2a4c97b640 10 monclient(hunting): picked mon.noname-b con 0x7f2a480b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:08.277 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.274+0000 7f2a4c97b640 10 monclient(hunting): picked mon.noname-c con 0x7f2a4814be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:08.277 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.274+0000 7f2a4c97b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a480af940 con 0x7f2a480b4d60 2024-03-25T18:28:08.277 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.274+0000 7f2a4c97b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a480b0a70 con 0x7f2a4814be90 2024-03-25T18:28:08.277 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.274+0000 7f2a4c97b640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a480b0d50 con 0x7f2a480b1730 2024-03-25T18:28:08.277 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.274+0000 7f2a4c97b640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:08.277 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.274+0000 7f2a4c97b640 10 monclient(hunting): authenticate will time out at 7506.442676s 2024-03-25T18:28:08.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.288+0000 7f2a46d9d640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f2a4814be90 0x7f2a4814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:55638/0 (socket says 172.21.15.99:55638) 2024-03-25T18:28:08.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.288+0000 7f2a46d9d640 1 -- 172.21.15.99:0/755916138 learned_addr learned my addr 172.21.15.99:0/755916138 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:08.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.301+0000 7f2a4559a640 1 -- 172.21.15.99:0/755916138 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298617256 0 0) 0x7f2a480b0a70 con 0x7f2a4814be90 2024-03-25T18:28:08.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.301+0000 7f2a4559a640 10 monclient(hunting): _negotiate 2024-03-25T18:28:08.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.301+0000 7f2a4559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:08.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.301+0000 7f2a4559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:08.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.301+0000 7f2a4559a640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:28:08.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.301+0000 7f2a4559a640 1 -- 172.21.15.99:0/755916138 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a30003520 con 0x7f2a4814be90 2024-03-25T18:28:08.317 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.316+0000 7f2a4559a640 1 -- 172.21.15.99:0/755916138 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865198396 0 0) 0x7f2a480af940 con 0x7f2a480b4d60 2024-03-25T18:28:08.317 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.316+0000 7f2a4559a640 10 monclient(hunting): _negotiate 2024-03-25T18:28:08.317 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.316+0000 7f2a4559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:08.317 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.316+0000 7f2a4559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:08.317 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.316+0000 7f2a4559a640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:28:08.317 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.316+0000 7f2a4559a640 1 -- 172.21.15.99:0/755916138 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a480b0a70 con 0x7f2a480b4d60 2024-03-25T18:28:08.324 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.323+0000 7f2a4559a640 1 -- 172.21.15.99:0/755916138 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865412931 0 0) 0x7f2a480b0d50 con 0x7f2a480b1730 2024-03-25T18:28:08.325 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.323+0000 7f2a4559a640 10 monclient(hunting): _negotiate 2024-03-25T18:28:08.325 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.323+0000 7f2a4559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:08.325 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.323+0000 7f2a4559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:08.325 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.323+0000 7f2a4559a640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:28:08.325 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.323+0000 7f2a4559a640 1 -- 172.21.15.99:0/755916138 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a480af940 con 0x7f2a480b1730 2024-03-25T18:28:08.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.330+0000 7f2a4559a640 1 -- 172.21.15.99:0/755916138 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 752422732 0 0) 0x7f2a30003520 con 0x7f2a4814be90 2024-03-25T18:28:08.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.330+0000 7f2a4559a640 1 -- 172.21.15.99:0/755916138 >> v1:172.21.15.99:6790/0 conn(0x7f2a480b1730 legacy=0x7f2a480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:08.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.330+0000 7f2a4559a640 1 -- 172.21.15.99:0/755916138 >> v1:172.21.15.99:6789/0 conn(0x7f2a480b4d60 legacy=0x7f2a4813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:08.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.330+0000 7f2a4559a640 10 monclient: _finish_hunting 0 2024-03-25T18:28:08.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.330+0000 7f2a4559a640 1 monclient: found mon.noname-c 2024-03-25T18:28:08.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.330+0000 7f2a4559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:08.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.330+0000 7f2a4559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:28:08.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.330+0000 7f2a4559a640 1 -- 172.21.15.99:0/755916138 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a480b1030 con 0x7f2a4814be90 2024-03-25T18:28:08.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.330+0000 7f2a4559a640 10 monclient: _finish_auth 0 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.330+0000 7f2a4559a640 10 monclient: _check_auth_tickets 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.330+0000 7f2a4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.330+0000 7f2a4559a640 1 -- 172.21.15.99:0/755916138 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2a40001a10 con 0x7f2a4814be90 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.330+0000 7f2a4559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.331+0000 7f2a4559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.331+0000 7f2a4559a640 10 monclient: dump: 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:08.333 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:08.334 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:08.334 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.331+0000 7f2a4c97b640 5 monclient: authenticate success, global_id 4345 2024-03-25T18:28:08.334 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.331+0000 7f2a4c97b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:08.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.343+0000 7f2a4559a640 1 -- 172.21.15.99:0/755916138 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a40001b70 con 0x7f2a4814be90 2024-03-25T18:28:08.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.343+0000 7f2a4559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:08.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.343+0000 7f2a4559a640 1 -- 172.21.15.99:0/755916138 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2a40001f80 con 0x7f2a4814be90 2024-03-25T18:28:08.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.343+0000 7f2a4c97b640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:08.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.343+0000 7f2a4c97b640 4 set_mon_vals no callback set 2024-03-25T18:28:08.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.343+0000 7f2a4559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:08.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.343+0000 7f2a4559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:08.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.343+0000 7f2a4559a640 10 monclient: dump: 2024-03-25T18:28:08.344 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:08.344 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:08.344 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:08.345 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:08.345 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:08.345 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:08.345 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:08.345 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:08.345 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:08.345 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:08.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.344+0000 7f2a4c97b640 10 monclient: shutdown 2024-03-25T18:28:08.347 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.344+0000 7f2a4c97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:08.347 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.344+0000 7f2a4c97b640 1 -- 172.21.15.99:0/755916138 >> v1:172.21.15.190:6789/0 conn(0x7f2a4814be90 legacy=0x7f2a4814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:08.347 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.344+0000 7f2a4c97b640 1 -- 172.21.15.99:0/755916138 shutdown_connections 2024-03-25T18:28:08.347 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.344+0000 7f2a4c97b640 1 -- 172.21.15.99:0/755916138 >> 172.21.15.99:0/755916138 conn(0x7f2a480617a0 msgr2=0x7f2a48061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:08.347 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.344+0000 7f2a4c97b640 1 -- 172.21.15.99:0/755916138 shutdown_connections 2024-03-25T18:28:08.347 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.344+0000 7f2a4c97b640 1 -- 172.21.15.99:0/755916138 wait complete. 2024-03-25T18:28:08.347 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.345+0000 7f2a4c97b640 10 monclient: build_initial_monmap 2024-03-25T18:28:08.347 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.345+0000 7f2a4c97b640 10 monclient: monmap: 2024-03-25T18:28:08.347 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:08.347 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:08.347 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:28:08.347 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:28:08.347 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:08.348 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:28:08.348 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:08.348 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:28:08.348 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:28:08.348 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:08.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.345+0000 7f2a4c97b640 1 librados: starting msgr at 2024-03-25T18:28:08.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.345+0000 7f2a4c97b640 1 librados: starting objecter 2024-03-25T18:28:08.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 1 Processor -- start 2024-03-25T18:28:08.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 1 -- start start 2024-03-25T18:28:08.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 1 librados: setting wanted keys 2024-03-25T18:28:08.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 1 librados: calling monclient init 2024-03-25T18:28:08.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 10 monclient: init 2024-03-25T18:28:08.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:08.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:08.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 10 monclient(hunting): picked mon.noname-c con 0x7f2a480b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:08.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 10 monclient(hunting): picked mon.noname-a con 0x7f2a480b4d60 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:08.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 10 monclient(hunting): picked mon.noname-b con 0x7f2a4814be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:08.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4813b940 con 0x7f2a480b4d60 2024-03-25T18:28:08.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a481d7cf0 con 0x7f2a4814be90 2024-03-25T18:28:08.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4813a9e0 con 0x7f2a480b1730 2024-03-25T18:28:08.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:08.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.346+0000 7f2a4c97b640 10 monclient(hunting): authenticate will time out at 7506.514630s 2024-03-25T18:28:08.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.354+0000 7f2a4659c640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f2a480b1730 0x7f2a481d8210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:36194/0 (socket says 172.21.15.99:36194) 2024-03-25T18:28:08.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.354+0000 7f2a4659c640 1 -- 172.21.15.99:0/1587203524 learned_addr learned my addr 172.21.15.99:0/1587203524 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:08.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.356+0000 7f2a36ffd640 10 client.?.objecter ms_handle_connect 0x7f2a480b1730 2024-03-25T18:28:08.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.356+0000 7f2a36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:08.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.366+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339437648 0 0) 0x7f2a4813a9e0 con 0x7f2a480b1730 2024-03-25T18:28:08.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.366+0000 7f2a36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:08.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.366+0000 7f2a36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:08.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.366+0000 7f2a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:08.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.366+0000 7f2a36ffd640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:28:08.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.366+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a28004eb0 con 0x7f2a480b1730 2024-03-25T18:28:08.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.368+0000 7f2a36ffd640 10 client.?.objecter ms_handle_connect 0x7f2a480b4d60 2024-03-25T18:28:08.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.368+0000 7f2a36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:08.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.371+0000 7f2a36ffd640 10 client.?.objecter ms_handle_connect 0x7f2a4814be90 2024-03-25T18:28:08.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.371+0000 7f2a36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:08.382 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.380+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275113484 0 0) 0x7f2a4813b940 con 0x7f2a480b4d60 2024-03-25T18:28:08.382 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.381+0000 7f2a36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:08.382 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.381+0000 7f2a36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:08.382 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.381+0000 7f2a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:08.382 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.381+0000 7f2a36ffd640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:28:08.382 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.381+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a4813a9e0 con 0x7f2a480b4d60 2024-03-25T18:28:08.387 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.386+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304457111 0 0) 0x7f2a481d7cf0 con 0x7f2a4814be90 2024-03-25T18:28:08.387 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.386+0000 7f2a36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:08.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.386+0000 7f2a36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:08.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.386+0000 7f2a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:08.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.386+0000 7f2a36ffd640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:28:08.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.386+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a4813b940 con 0x7f2a4814be90 2024-03-25T18:28:08.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.390+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3899428153 0 0) 0x7f2a28004eb0 con 0x7f2a480b1730 2024-03-25T18:28:08.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.390+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a481d7cf0 con 0x7f2a480b1730 2024-03-25T18:28:08.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.390+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2a38001a30 con 0x7f2a480b1730 2024-03-25T18:28:08.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.390+0000 7f2a36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:08.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.390+0000 7f2a36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:08.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.390+0000 7f2a36ffd640 10 monclient(hunting): dump: 2024-03-25T18:28:08.392 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:08.392 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:08.392 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:08.392 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:08.392 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:08.392 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:08.392 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:08.392 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:08.393 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:08.393 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:08.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.390+0000 7f2a36ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:08.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.391+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4094015180 0 0) 0x7f2a4813a9e0 con 0x7f2a480b4d60 2024-03-25T18:28:08.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.391+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a28004eb0 con 0x7f2a480b4d60 2024-03-25T18:28:08.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.391+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2a3c001870 con 0x7f2a480b4d60 2024-03-25T18:28:08.394 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.391+0000 7f2a36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:08.394 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.391+0000 7f2a36ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:08.394 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.391+0000 7f2a36ffd640 10 monclient(hunting): dump: 2024-03-25T18:28:08.394 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:08.394 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:08.394 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:08.394 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:08.394 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:08.394 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:08.394 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:08.394 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:08.394 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:08.394 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:08.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2466103211 0 0) 0x7f2a28004eb0 con 0x7f2a480b4d60 2024-03-25T18:28:08.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 >> v1:172.21.15.99:6790/0 conn(0x7f2a480b1730 legacy=0x7f2a481d8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:08.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 >> v1:172.21.15.190:6789/0 conn(0x7f2a4814be90 legacy=0x7f2a48138020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:08.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a36ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:08.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a36ffd640 1 monclient: found mon.a 2024-03-25T18:28:08.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:08.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a36ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:08.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a481d7910 con 0x7f2a480b4d60 2024-03-25T18:28:08.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a36ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:08.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a36ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:08.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:08.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a4c97b640 5 monclient: authenticate success, global_id 4332 2024-03-25T18:28:08.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a4c97b640 10 monclient: _renew_subs 2024-03-25T18:28:08.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a4c97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:08.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a4c97b640 1 -- 172.21.15.99:0/1587203524 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a48139c80 con 0x7f2a480b4d60 2024-03-25T18:28:08.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a4c97b640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:08.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a4c97b640 10 monclient: _renew_subs 2024-03-25T18:28:08.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a4c97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:08.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a4c97b640 1 -- 172.21.15.99:0/1587203524 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a48138910 con 0x7f2a480b4d60 2024-03-25T18:28:08.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.403+0000 7f2a4c97b640 1 librados: init done 2024-03-25T18:28:08.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.404+0000 7f2a4c97b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:08.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.404+0000 7f2a4c97b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:08.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.404+0000 7f2a4c97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:08.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.404+0000 7f2a4c97b640 1 -- 172.21.15.99:0/1587203524 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a481d7b00 con 0x7f2a480b4d60 2024-03-25T18:28:08.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.732+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a3c0035f0 con 0x7f2a480b4d60 2024-03-25T18:28:08.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.732+0000 7f2a36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:08.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.732+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2a3c001d80 con 0x7f2a480b4d60 2024-03-25T18:28:08.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.732+0000 7f2a36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:08.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.732+0000 7f2a36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:08.734 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.732+0000 7f2a36ffd640 10 monclient: dump: 2024-03-25T18:28:08.734 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:08.734 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:08.734 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:08.734 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:08.734 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:08.734 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:08.734 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:08.734 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:08.734 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:08.734 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:08.734 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.732+0000 7f2a44d99640 4 set_mon_vals no callback set 2024-03-25T18:28:08.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.733+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f2a3c003b00 con 0x7f2a480b4d60 2024-03-25T18:28:08.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.733+0000 7f2a4659c640 10 monclient: get_auth_request con 0x7f2a2806dfb0 auth_method 0 2024-03-25T18:28:08.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.734+0000 7f2a36ffd640 10 client.4332.objecter ms_handle_connect 0x7f2a2806dfb0 2024-03-25T18:28:08.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.738+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f2a3c053570 con 0x7f2a480b4d60 2024-03-25T18:28:08.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.738+0000 7f2a36ffd640 10 client.4332.objecter ms_dispatch 0x7f2a48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:08.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.738+0000 7f2a36ffd640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:08.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.738+0000 7f2a36ffd640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:08.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.738+0000 7f2a36ffd640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-25T18:28:08.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.738+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a3c04f080 con 0x7f2a480b4d60 2024-03-25T18:28:08.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.738+0000 7f2a36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:08.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.738+0000 7f2a36ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:08.835 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:03:09.428 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:08.892 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:03:09.485 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:08.932 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:03:09.525 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:08.945 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:03:09.504 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:08.958 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:03:09.517 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:08.988 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.987+0000 7f2a4c97b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:08.988 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.987+0000 7f2a4c97b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:08.988 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.987+0000 7f2a4c97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:08.988 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:08.987+0000 7f2a4c97b640 1 -- 172.21.15.99:0/1587203524 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a480627b0 con 0x7f2a480b4d60 2024-03-25T18:28:08.990 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:03:09.583 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:08.993 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:03:09.551 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:09.002 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:03:09.595 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:09.030 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:03:09.623 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:09.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.070+0000 7f2a36ffd640 1 -- 172.21.15.99:0/1587203524 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f2a3c050fb0 con 0x7f2a480b4d60 2024-03-25T18:28:09.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.070+0000 7f2a36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:09.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.070+0000 7f2a36ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:09.072 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:28:09.072 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:09.075 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.073+0000 7f2a4c97b640 10 librados: watch_flush enter 2024-03-25T18:28:09.075 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.073+0000 7f2a4c97b640 10 librados: watch_flush exit 2024-03-25T18:28:09.075 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.073+0000 7f2a4c97b640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-25T18:28:09.075 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.073+0000 7f2a4c97b640 10 client.4332.objecter successfully canceled tick 2024-03-25T18:28:09.075 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.073+0000 7f2a4c97b640 1 -- 172.21.15.99:0/1587203524 >> v1:172.21.15.190:6800/4220409699 conn(0x7f2a2806dfb0 legacy=0x7f2a28070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:09.075 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.073+0000 7f2a4c97b640 10 monclient: shutdown 2024-03-25T18:28:09.075 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.073+0000 7f2a4c97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:09.075 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.073+0000 7f2a4c97b640 1 -- 172.21.15.99:0/1587203524 >> v1:172.21.15.99:6789/0 conn(0x7f2a480b4d60 legacy=0x7f2a4813a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:09.075 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.073+0000 7f2a4c97b640 1 -- 172.21.15.99:0/1587203524 shutdown_connections 2024-03-25T18:28:09.075 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.073+0000 7f2a4c97b640 1 -- 172.21.15.99:0/1587203524 >> 172.21.15.99:0/1587203524 conn(0x7f2a480617a0 msgr2=0x7f2a480b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:09.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.073+0000 7f2a4c97b640 1 -- 172.21.15.99:0/1587203524 shutdown_connections 2024-03-25T18:28:09.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.073+0000 7f2a4c97b640 1 -- 172.21.15.99:0/1587203524 wait complete. 2024-03-25T18:28:09.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:09.073+0000 7f2a4c97b640 1 librados: shutdown 2024-03-25T18:28:09.088 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:15.896 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:16.003 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.002+0000 7f6ceb83e640 10 monclient: get_monmap_and_config 2024-03-25T18:28:16.003 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.002+0000 7f6ceb83e640 10 monclient: build_initial_monmap 2024-03-25T18:28:16.003 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.002+0000 7f6ceb83e640 10 monclient: monmap: 2024-03-25T18:28:16.003 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:16.003 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:16.003 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:28:16.003454+0000 2024-03-25T18:28:16.003 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:28:16.003454+0000 2024-03-25T18:28:16.003 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:16.004 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:16.004 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:16.004 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:28:16.004 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:28:16.004 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:16.005 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.002+0000 7f6ceb83e640 1 Processor -- start 2024-03-25T18:28:16.005 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.003+0000 7f6ceb83e640 1 -- start start 2024-03-25T18:28:16.005 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.003+0000 7f6ceb83e640 10 monclient: init 2024-03-25T18:28:16.005 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.003+0000 7f6ceb83e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:16.005 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.003+0000 7f6ceb83e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:16.006 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.003+0000 7f6ceb83e640 10 monclient(hunting): picked mon.noname-b con 0x7f6ce40b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:16.006 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.003+0000 7f6ceb83e640 10 monclient(hunting): picked mon.noname-a con 0x7f6ce40b1730 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:16.006 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.003+0000 7f6ceb83e640 10 monclient(hunting): picked mon.noname-c con 0x7f6ce414be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:16.006 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.003+0000 7f6ceb83e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce40af940 con 0x7f6ce40b1730 2024-03-25T18:28:16.006 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.003+0000 7f6ceb83e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce40b0a70 con 0x7f6ce414be90 2024-03-25T18:28:16.006 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.003+0000 7f6ceb83e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce40b0d50 con 0x7f6ce40b4d60 2024-03-25T18:28:16.006 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.003+0000 7f6ceb83e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:16.006 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.003+0000 7f6ceb83e640 10 monclient(hunting): authenticate will time out at 7514.171612s 2024-03-25T18:28:16.006 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.004+0000 7f6ce95da640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f6ce40b4d60 0x7f6ce413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:56270/0 (socket says 172.21.15.99:56270) 2024-03-25T18:28:16.006 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.004+0000 7f6ce95da640 1 -- 172.21.15.99:0/4221450607 learned_addr learned my addr 172.21.15.99:0/4221450607 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:16.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.050+0000 7f6cdbfff640 1 -- 172.21.15.99:0/4221450607 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423794868 0 0) 0x7f6ce40af940 con 0x7f6ce40b1730 2024-03-25T18:28:16.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.050+0000 7f6cdbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.050+0000 7f6cdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.055 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.050+0000 7f6cdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.055 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.050+0000 7f6cdbfff640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:28:16.055 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.050+0000 7f6cdbfff640 1 -- 172.21.15.99:0/4221450607 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cd0003520 con 0x7f6ce40b1730 2024-03-25T18:28:16.077 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 1 -- 172.21.15.99:0/4221450607 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3190183089 0 0) 0x7f6cd0003520 con 0x7f6ce40b1730 2024-03-25T18:28:16.077 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 1 -- 172.21.15.99:0/4221450607 >> v1:172.21.15.99:6790/0 conn(0x7f6ce40b4d60 legacy=0x7f6ce413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.077 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 1 -- 172.21.15.99:0/4221450607 >> v1:172.21.15.190:6789/0 conn(0x7f6ce414be90 legacy=0x7f6ce414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.077 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:16.077 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 1 monclient: found mon.noname-a 2024-03-25T18:28:16.077 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 1 -- 172.21.15.99:0/4221450607 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce40b1030 con 0x7f6ce40b1730 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 10 monclient: _finish_auth 0 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 1 -- 172.21.15.99:0/4221450607 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6cd4001690 con 0x7f6ce40b1730 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6ceb83e640 5 monclient: authenticate success, global_id 4335 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6ceb83e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6cdbfff640 10 monclient: dump: 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:16.078 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:16.079 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.079 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:16.079 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:16.079 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:16.079 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:16.079 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:16.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.076+0000 7f6ceb83e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:16.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.087+0000 7f6cdbfff640 1 -- 172.21.15.99:0/4221450607 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cd40017f0 con 0x7f6ce40b1730 2024-03-25T18:28:16.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.087+0000 7f6cdbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:16.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.087+0000 7f6cdbfff640 1 -- 172.21.15.99:0/4221450607 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6cd4001c00 con 0x7f6ce40b1730 2024-03-25T18:28:16.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.087+0000 7f6cdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.087+0000 7f6cdbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:16.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.087+0000 7f6cdbfff640 10 monclient: dump: 2024-03-25T18:28:16.089 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:16.089 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:16.089 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:16.089 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:16.089 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.089 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:16.089 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:16.089 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:16.089 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:16.089 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:16.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.087+0000 7f6ceb83e640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:16.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.087+0000 7f6ceb83e640 4 set_mon_vals no callback set 2024-03-25T18:28:16.090 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.088+0000 7f6ceb83e640 10 monclient: shutdown 2024-03-25T18:28:16.090 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.088+0000 7f6ceb83e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:16.090 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.088+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4221450607 >> v1:172.21.15.99:6789/0 conn(0x7f6ce40b1730 legacy=0x7f6ce40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.088+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4221450607 shutdown_connections 2024-03-25T18:28:16.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.088+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4221450607 >> 172.21.15.99:0/4221450607 conn(0x7f6ce40617a0 msgr2=0x7f6ce4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:16.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.089+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4221450607 shutdown_connections 2024-03-25T18:28:16.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.089+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4221450607 wait complete. 2024-03-25T18:28:16.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.089+0000 7f6ceb83e640 10 monclient: build_initial_monmap 2024-03-25T18:28:16.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.089+0000 7f6ceb83e640 10 monclient: monmap: 2024-03-25T18:28:16.091 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:16.091 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:16.091 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:28:16.091 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:28:16.091 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:16.091 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:28:16.091 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:16.091 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:28:16.092 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:28:16.092 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:16.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.090+0000 7f6ceb83e640 1 librados: starting msgr at 2024-03-25T18:28:16.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.090+0000 7f6ceb83e640 1 librados: starting objecter 2024-03-25T18:28:16.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.090+0000 7f6ceb83e640 1 Processor -- start 2024-03-25T18:28:16.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.090+0000 7f6ceb83e640 1 -- start start 2024-03-25T18:28:16.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.090+0000 7f6ceb83e640 1 librados: setting wanted keys 2024-03-25T18:28:16.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.090+0000 7f6ceb83e640 1 librados: calling monclient init 2024-03-25T18:28:16.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.090+0000 7f6ceb83e640 10 monclient: init 2024-03-25T18:28:16.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.090+0000 7f6ceb83e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:16.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.090+0000 7f6ceb83e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:16.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.090+0000 7f6ceb83e640 10 monclient(hunting): picked mon.noname-b con 0x7f6ce40b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:16.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.090+0000 7f6ceb83e640 10 monclient(hunting): picked mon.noname-c con 0x7f6ce40b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:16.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.091+0000 7f6ceb83e640 10 monclient(hunting): picked mon.noname-a con 0x7f6ce414be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:16.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.091+0000 7f6ceb83e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce4140380 con 0x7f6ce414be90 2024-03-25T18:28:16.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.091+0000 7f6ceb83e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce4141ca0 con 0x7f6ce40b1730 2024-03-25T18:28:16.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.091+0000 7f6ceb83e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce4140860 con 0x7f6ce40b4d60 2024-03-25T18:28:16.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.091+0000 7f6ceb83e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:16.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.091+0000 7f6ceb83e640 10 monclient(hunting): authenticate will time out at 7514.259053s 2024-03-25T18:28:16.103 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.102+0000 7f6ce9ddb640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f6ce414be90 0x7f6ce41e22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:33140/0 (socket says 172.21.15.99:33140) 2024-03-25T18:28:16.103 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.102+0000 7f6ce9ddb640 1 -- 172.21.15.99:0/4220642306 learned_addr learned my addr 172.21.15.99:0/4220642306 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:16.106 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.105+0000 7f6cd9ffb640 10 client.?.objecter ms_handle_connect 0x7f6ce414be90 2024-03-25T18:28:16.106 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.105+0000 7f6cd9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:16.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.113+0000 7f6cd9ffb640 10 client.?.objecter ms_handle_connect 0x7f6ce40b1730 2024-03-25T18:28:16.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.113+0000 7f6cd9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:16.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.115+0000 7f6cd9ffb640 10 client.?.objecter ms_handle_connect 0x7f6ce40b4d60 2024-03-25T18:28:16.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.115+0000 7f6cd9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:16.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.116+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900850010 0 0) 0x7f6ce4140380 con 0x7f6ce414be90 2024-03-25T18:28:16.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.116+0000 7f6cd9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.116+0000 7f6cd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.118 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.116+0000 7f6cd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.118 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.116+0000 7f6cd9ffb640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:28:16.118 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.116+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc8004eb0 con 0x7f6ce414be90 2024-03-25T18:28:16.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.127+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470070870 0 0) 0x7f6ce4140860 con 0x7f6ce40b4d60 2024-03-25T18:28:16.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.127+0000 7f6cd9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.127+0000 7f6cd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.127+0000 7f6cd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.127+0000 7f6cd9ffb640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:28:16.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.127+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ce4140380 con 0x7f6ce40b4d60 2024-03-25T18:28:16.130 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.129+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548461224 0 0) 0x7f6ce4141ca0 con 0x7f6ce40b1730 2024-03-25T18:28:16.130 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.129+0000 7f6cd9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:16.130 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.129+0000 7f6cd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:16.130 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.129+0000 7f6cd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:16.131 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.129+0000 7f6cd9ffb640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:28:16.131 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.129+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ce4140860 con 0x7f6ce40b1730 2024-03-25T18:28:16.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.137+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3347921286 0 0) 0x7f6cc8004eb0 con 0x7f6ce414be90 2024-03-25T18:28:16.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.137+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ce4141ca0 con 0x7f6ce414be90 2024-03-25T18:28:16.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.137+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6cdc001c30 con 0x7f6ce414be90 2024-03-25T18:28:16.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.137+0000 7f6cd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.137+0000 7f6cd9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:16.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.137+0000 7f6cd9ffb640 10 monclient(hunting): dump: 2024-03-25T18:28:16.138 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:16.138 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:16.138 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:16.139 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:16.139 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.139 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:16.139 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:16.139 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:16.139 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:16.139 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:16.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.137+0000 7f6cd9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:16.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.148+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2133221922 0 0) 0x7f6ce4140380 con 0x7f6ce40b4d60 2024-03-25T18:28:16.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.149+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cc8004eb0 con 0x7f6ce40b4d60 2024-03-25T18:28:16.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.149+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6cd4001b80 con 0x7f6ce40b4d60 2024-03-25T18:28:16.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.149+0000 7f6cd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.149+0000 7f6cd9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:16.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.149+0000 7f6cd9ffb640 10 monclient(hunting): dump: 2024-03-25T18:28:16.150 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:16.151 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:16.151 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:16.151 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:16.151 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.151 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:16.151 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:16.151 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:16.151 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:16.151 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:16.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.149+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3486208221 0 0) 0x7f6ce4140860 con 0x7f6ce40b1730 2024-03-25T18:28:16.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.149+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ce4140380 con 0x7f6ce40b1730 2024-03-25T18:28:16.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.149+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6ce0001860 con 0x7f6ce40b1730 2024-03-25T18:28:16.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.149+0000 7f6cd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.149+0000 7f6cd9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:16.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.149+0000 7f6cd9ffb640 10 monclient(hunting): dump: 2024-03-25T18:28:16.152 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:16.152 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:16.152 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:16.152 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:16.152 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.152 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:16.152 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:16.152 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:16.152 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:16.152 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:16.153 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4142812912 0 0) 0x7f6ce4141ca0 con 0x7f6ce414be90 2024-03-25T18:28:16.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 >> v1:172.21.15.99:6790/0 conn(0x7f6ce40b4d60 legacy=0x7f6ce41db720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 >> v1:172.21.15.190:6789/0 conn(0x7f6ce40b1730 legacy=0x7f6ce41db010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6cd9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:28:16.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6cd9ffb640 1 monclient: found mon.a 2024-03-25T18:28:16.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6cd9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:16.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6cd9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:16.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce41423a0 con 0x7f6ce414be90 2024-03-25T18:28:16.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6cd9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:28:16.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6cd9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:28:16.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6cd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:16.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6ceb83e640 5 monclient: authenticate success, global_id 4338 2024-03-25T18:28:16.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6ceb83e640 10 monclient: _renew_subs 2024-03-25T18:28:16.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6ceb83e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:16.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4220642306 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce4138650 con 0x7f6ce414be90 2024-03-25T18:28:16.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6ceb83e640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:16.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6ceb83e640 10 monclient: _renew_subs 2024-03-25T18:28:16.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6ceb83e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:16.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4220642306 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce4140d00 con 0x7f6ce414be90 2024-03-25T18:28:16.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.151+0000 7f6ceb83e640 1 librados: init done 2024-03-25T18:28:16.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.152+0000 7f6ceb83e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:16.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.152+0000 7f6ceb83e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:16.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.152+0000 7f6ceb83e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:16.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.152+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4220642306 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce40627b0 con 0x7f6ce414be90 2024-03-25T18:28:16.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.452+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cdc003990 con 0x7f6ce414be90 2024-03-25T18:28:16.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.452+0000 7f6cd9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:16.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.452+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6cdc003da0 con 0x7f6ce414be90 2024-03-25T18:28:16.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.452+0000 7f6cd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:16.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.452+0000 7f6cd9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:16.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.452+0000 7f6cd9ffb640 10 monclient: dump: 2024-03-25T18:28:16.454 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:16.454 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:16.454 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:16.454 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:16.454 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:16.454 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:16.454 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:16.454 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:16.454 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:16.454 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:16.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.452+0000 7f6cdb7fe640 4 set_mon_vals no callback set 2024-03-25T18:28:16.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.453+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f6cdc003fc0 con 0x7f6ce414be90 2024-03-25T18:28:16.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.453+0000 7f6ce95da640 10 monclient: get_auth_request con 0x7f6cc806e0f0 auth_method 0 2024-03-25T18:28:16.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.453+0000 7f6cd9ffb640 10 client.4338.objecter ms_handle_connect 0x7f6cc806e0f0 2024-03-25T18:28:16.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.462+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f6cdc054980 con 0x7f6ce414be90 2024-03-25T18:28:16.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.462+0000 7f6cd9ffb640 10 client.4338.objecter ms_dispatch 0x7f6ce4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:16.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.462+0000 7f6cd9ffb640 3 client.4338.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:16.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.462+0000 7f6cd9ffb640 3 client.4338.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:16.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.462+0000 7f6cd9ffb640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-25T18:28:16.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.462+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6cdc050490 con 0x7f6ce414be90 2024-03-25T18:28:16.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.462+0000 7f6cd9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:16.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.462+0000 7f6cd9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:16.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.711+0000 7f6ceb83e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:16.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.711+0000 7f6ceb83e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:16.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.711+0000 7f6ceb83e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:16.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.711+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4220642306 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ce4069f40 con 0x7f6ce414be90 2024-03-25T18:28:16.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.736+0000 7f6cd9ffb640 1 -- 172.21.15.99:0/4220642306 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f6cdc0523c0 con 0x7f6ce414be90 2024-03-25T18:28:16.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.736+0000 7f6cd9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:16.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.736+0000 7f6cd9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:16.737 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:28:16.737 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:16.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.738+0000 7f6ceb83e640 10 librados: watch_flush enter 2024-03-25T18:28:16.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.738+0000 7f6ceb83e640 10 librados: watch_flush exit 2024-03-25T18:28:16.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.738+0000 7f6ceb83e640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-25T18:28:16.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.738+0000 7f6ceb83e640 10 client.4338.objecter successfully canceled tick 2024-03-25T18:28:16.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.738+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4220642306 >> v1:172.21.15.190:6800/4220409699 conn(0x7f6cc806e0f0 legacy=0x7f6cc80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.738+0000 7f6ceb83e640 10 monclient: shutdown 2024-03-25T18:28:16.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.738+0000 7f6ceb83e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:16.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.738+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4220642306 >> v1:172.21.15.99:6789/0 conn(0x7f6ce414be90 legacy=0x7f6ce41e22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:16.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.738+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4220642306 shutdown_connections 2024-03-25T18:28:16.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.738+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4220642306 >> 172.21.15.99:0/4220642306 conn(0x7f6ce40617a0 msgr2=0x7f6ce413c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:16.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.738+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4220642306 shutdown_connections 2024-03-25T18:28:16.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.738+0000 7f6ceb83e640 1 -- 172.21.15.99:0/4220642306 wait complete. 2024-03-25T18:28:16.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:16.738+0000 7f6ceb83e640 1 librados: shutdown 2024-03-25T18:28:16.751 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:23.559 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:23.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.672+0000 7f02721c6640 10 monclient: get_monmap_and_config 2024-03-25T18:28:23.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.672+0000 7f02721c6640 10 monclient: build_initial_monmap 2024-03-25T18:28:23.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.673+0000 7f02721c6640 10 monclient: monmap: 2024-03-25T18:28:23.674 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:23.674 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:23.674 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:28:23.674038+0000 2024-03-25T18:28:23.674 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:28:23.674038+0000 2024-03-25T18:28:23.674 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:23.674 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:23.675 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:23.675 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:28:23.675 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:28:23.675 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:23.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.674+0000 7f02721c6640 1 Processor -- start 2024-03-25T18:28:23.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.674+0000 7f02721c6640 1 -- start start 2024-03-25T18:28:23.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.674+0000 7f02721c6640 10 monclient: init 2024-03-25T18:28:23.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.675+0000 7f02721c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:23.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.675+0000 7f02721c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:23.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.675+0000 7f02721c6640 10 monclient(hunting): picked mon.noname-b con 0x7f026c0a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:23.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.675+0000 7f02721c6640 10 monclient(hunting): picked mon.noname-c con 0x7f026c13f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:23.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.675+0000 7f02721c6640 10 monclient(hunting): picked mon.noname-a con 0x7f026c1430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:23.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.675+0000 7f02721c6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c142520 con 0x7f026c1430c0 2024-03-25T18:28:23.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.675+0000 7f02721c6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c0b28a0 con 0x7f026c13f710 2024-03-25T18:28:23.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.675+0000 7f02721c6640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c0b2a30 con 0x7f026c0a98f0 2024-03-25T18:28:23.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.675+0000 7f02721c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:23.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.675+0000 7f02721c6640 10 monclient(hunting): authenticate will time out at 7521.843307s 2024-03-25T18:28:23.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.676+0000 7f026b7fe640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f026c0a98f0 0x7f026c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:32800/0 (socket says 172.21.15.99:32800) 2024-03-25T18:28:23.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.676+0000 7f026b7fe640 1 -- 172.21.15.99:0/1489632807 learned_addr learned my addr 172.21.15.99:0/1489632807 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:23.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.684+0000 7f026affd640 1 -- 172.21.15.99:0/1489632807 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997232228 0 0) 0x7f026c0b2a30 con 0x7f026c0a98f0 2024-03-25T18:28:23.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.684+0000 7f026affd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:23.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.684+0000 7f026affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:23.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.684+0000 7f026affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:23.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.684+0000 7f026affd640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:28:23.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.684+0000 7f026affd640 1 -- 172.21.15.99:0/1489632807 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0258003520 con 0x7f026c0a98f0 2024-03-25T18:28:23.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.694+0000 7f026affd640 1 -- 172.21.15.99:0/1489632807 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574124785 0 0) 0x7f026c142520 con 0x7f026c1430c0 2024-03-25T18:28:23.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.694+0000 7f026affd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:23.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.694+0000 7f026affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:23.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.694+0000 7f026affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:23.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.694+0000 7f026affd640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:28:23.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.694+0000 7f026affd640 1 -- 172.21.15.99:0/1489632807 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f026c0b2a30 con 0x7f026c1430c0 2024-03-25T18:28:23.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.695+0000 7f026affd640 1 -- 172.21.15.99:0/1489632807 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1753046177 0 0) 0x7f0258003520 con 0x7f026c0a98f0 2024-03-25T18:28:23.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.695+0000 7f026affd640 1 -- 172.21.15.99:0/1489632807 >> v1:172.21.15.190:6789/0 conn(0x7f026c13f710 legacy=0x7f026c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:23.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.695+0000 7f026affd640 1 -- 172.21.15.99:0/1489632807 >> v1:172.21.15.99:6789/0 conn(0x7f026c1430c0 legacy=0x7f026c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:23.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.695+0000 7f026affd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:23.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.695+0000 7f026affd640 1 monclient: found mon.noname-b 2024-03-25T18:28:23.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.695+0000 7f026affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:23.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.695+0000 7f026affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:28:23.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.695+0000 7f026affd640 1 -- 172.21.15.99:0/1489632807 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c0b44c0 con 0x7f026c0a98f0 2024-03-25T18:28:23.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.695+0000 7f026affd640 10 monclient: _finish_auth 0 2024-03-25T18:28:23.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.695+0000 7f026affd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:23.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.695+0000 7f026affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:23.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.695+0000 7f02721c6640 5 monclient: authenticate success, global_id 4349 2024-03-25T18:28:23.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.695+0000 7f02721c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:23.698 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.696+0000 7f026affd640 1 -- 172.21.15.99:0/1489632807 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f025c001650 con 0x7f026c0a98f0 2024-03-25T18:28:23.698 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.696+0000 7f026affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:23.698 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.696+0000 7f026affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:23.698 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.696+0000 7f026affd640 10 monclient: dump: 2024-03-25T18:28:23.698 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:23.698 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:23.698 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:23.698 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:23.698 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:23.698 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:23.698 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:23.699 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:23.699 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:23.699 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:23.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.696+0000 7f02721c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:23.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.708+0000 7f026affd640 1 -- 172.21.15.99:0/1489632807 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f025c0018a0 con 0x7f026c0a98f0 2024-03-25T18:28:23.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.708+0000 7f026affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:23.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.708+0000 7f026affd640 1 -- 172.21.15.99:0/1489632807 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f025c001c30 con 0x7f026c0a98f0 2024-03-25T18:28:23.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.708+0000 7f026affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:23.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.708+0000 7f026affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:23.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.708+0000 7f026affd640 10 monclient: dump: 2024-03-25T18:28:23.709 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:23.710 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:23.710 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:23.710 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:23.710 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:23.710 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:23.710 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:23.710 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:23.710 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:23.710 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:23.710 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.708+0000 7f02721c6640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:23.710 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.708+0000 7f02721c6640 4 set_mon_vals no callback set 2024-03-25T18:28:23.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.709+0000 7f02721c6640 10 monclient: shutdown 2024-03-25T18:28:23.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.709+0000 7f02721c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:23.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.709+0000 7f02721c6640 1 -- 172.21.15.99:0/1489632807 >> v1:172.21.15.99:6790/0 conn(0x7f026c0a98f0 legacy=0x7f026c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:23.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.709+0000 7f02721c6640 1 -- 172.21.15.99:0/1489632807 shutdown_connections 2024-03-25T18:28:23.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.709+0000 7f02721c6640 1 -- 172.21.15.99:0/1489632807 >> 172.21.15.99:0/1489632807 conn(0x7f026c0b7ea0 msgr2=0x7f026c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:23.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.709+0000 7f02721c6640 1 -- 172.21.15.99:0/1489632807 shutdown_connections 2024-03-25T18:28:23.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.709+0000 7f02721c6640 1 -- 172.21.15.99:0/1489632807 wait complete. 2024-03-25T18:28:23.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.710+0000 7f02721c6640 10 monclient: build_initial_monmap 2024-03-25T18:28:23.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.710+0000 7f02721c6640 10 monclient: monmap: 2024-03-25T18:28:23.713 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:23.713 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:23.713 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:28:23.713 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:28:23.713 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:23.713 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:28:23.713 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:23.713 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:28:23.714 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:28:23.714 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:23.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.710+0000 7f02721c6640 1 librados: starting msgr at 2024-03-25T18:28:23.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.710+0000 7f02721c6640 1 librados: starting objecter 2024-03-25T18:28:23.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.710+0000 7f02721c6640 1 Processor -- start 2024-03-25T18:28:23.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.710+0000 7f02721c6640 1 -- start start 2024-03-25T18:28:23.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.710+0000 7f02721c6640 1 librados: setting wanted keys 2024-03-25T18:28:23.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.710+0000 7f02721c6640 1 librados: calling monclient init 2024-03-25T18:28:23.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.710+0000 7f02721c6640 10 monclient: init 2024-03-25T18:28:23.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.711+0000 7f02721c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:23.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.711+0000 7f02721c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:23.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.711+0000 7f02721c6640 10 monclient(hunting): picked mon.noname-b con 0x7f026c0a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:23.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.711+0000 7f02721c6640 10 monclient(hunting): picked mon.noname-a con 0x7f026c13f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:23.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.711+0000 7f02721c6640 10 monclient(hunting): picked mon.noname-c con 0x7f026c1430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:23.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.711+0000 7f02721c6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c136ec0 con 0x7f026c13f710 2024-03-25T18:28:23.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.711+0000 7f02721c6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c1355f0 con 0x7f026c0a98f0 2024-03-25T18:28:23.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.711+0000 7f02721c6640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c1319b0 con 0x7f026c1430c0 2024-03-25T18:28:23.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.711+0000 7f02721c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:23.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.711+0000 7f02721c6640 10 monclient(hunting): authenticate will time out at 7521.879141s 2024-03-25T18:28:23.720 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.719+0000 7f026bfff640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f026c1430c0 0x7f026c1312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:32822/0 (socket says 172.21.15.99:32822) 2024-03-25T18:28:23.720 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.719+0000 7f026bfff640 1 -- 172.21.15.99:0/1137035740 learned_addr learned my addr 172.21.15.99:0/1137035740 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:23.721 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.720+0000 7f0268ff9640 10 client.?.objecter ms_handle_connect 0x7f026c1430c0 2024-03-25T18:28:23.721 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.720+0000 7f0268ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:23.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.727+0000 7f0268ff9640 10 client.?.objecter ms_handle_connect 0x7f026c13f710 2024-03-25T18:28:23.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.727+0000 7f0268ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:23.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.727+0000 7f0268ff9640 10 client.?.objecter ms_handle_connect 0x7f026c0a98f0 2024-03-25T18:28:23.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.727+0000 7f0268ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:23.731 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.730+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917079551 0 0) 0x7f026c1319b0 con 0x7f026c1430c0 2024-03-25T18:28:23.731 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.730+0000 7f0268ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:23.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.730+0000 7f0268ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:23.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.730+0000 7f0268ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:23.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.730+0000 7f0268ff9640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:28:23.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.730+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f024c004eb0 con 0x7f026c1430c0 2024-03-25T18:28:23.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.736+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697661634 0 0) 0x7f026c136ec0 con 0x7f026c13f710 2024-03-25T18:28:23.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.736+0000 7f0268ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:23.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.736+0000 7f0268ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:23.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.736+0000 7f0268ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:23.738 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.736+0000 7f0268ff9640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:28:23.738 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.736+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f026c1319b0 con 0x7f026c13f710 2024-03-25T18:28:23.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.746+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599389401 0 0) 0x7f026c1355f0 con 0x7f026c0a98f0 2024-03-25T18:28:23.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.746+0000 7f0268ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:23.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.746+0000 7f0268ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:23.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.746+0000 7f0268ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:23.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.746+0000 7f0268ff9640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:28:23.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.746+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f026c136ec0 con 0x7f026c0a98f0 2024-03-25T18:28:23.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.751+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1941367013 0 0) 0x7f024c004eb0 con 0x7f026c1430c0 2024-03-25T18:28:23.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.751+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f026c1355f0 con 0x7f026c1430c0 2024-03-25T18:28:23.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.751+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0264001c90 con 0x7f026c1430c0 2024-03-25T18:28:23.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.751+0000 7f0268ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:23.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.751+0000 7f0268ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:23.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.751+0000 7f0268ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:23.753 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:23.753 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:23.753 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:23.753 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:23.753 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:23.753 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:23.753 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:23.753 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:23.753 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:23.754 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:23.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.751+0000 7f0268ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:23.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.762+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 459457 0 0) 0x7f026c1319b0 con 0x7f026c13f710 2024-03-25T18:28:23.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.762+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f024c004eb0 con 0x7f026c13f710 2024-03-25T18:28:23.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.762+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f02540018a0 con 0x7f026c13f710 2024-03-25T18:28:23.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.762+0000 7f0268ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:23.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.762+0000 7f0268ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:23.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.762+0000 7f0268ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:23.764 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:23.764 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:23.764 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:23.764 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:23.764 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:23.764 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:23.764 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:23.764 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:23.765 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:23.765 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:23.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.775+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1083930809 0 0) 0x7f026c136ec0 con 0x7f026c0a98f0 2024-03-25T18:28:23.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.775+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f026c1319b0 con 0x7f026c0a98f0 2024-03-25T18:28:23.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.775+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f025c001bb0 con 0x7f026c0a98f0 2024-03-25T18:28:23.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.775+0000 7f0268ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:23.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.775+0000 7f0268ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:23.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.775+0000 7f0268ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:23.776 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:23.776 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:23.777 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:23.777 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:23.777 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:23.777 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:23.777 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:23.777 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:23.777 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:23.777 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:23.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3343643598 0 0) 0x7f026c1355f0 con 0x7f026c1430c0 2024-03-25T18:28:23.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 >> v1:172.21.15.190:6789/0 conn(0x7f026c0a98f0 legacy=0x7f026c1dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:23.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 >> v1:172.21.15.99:6789/0 conn(0x7f026c13f710 legacy=0x7f026c130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:23.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f0268ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:23.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f0268ff9640 1 monclient: found mon.c 2024-03-25T18:28:23.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f0268ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:23.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f0268ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:28:23.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c135d80 con 0x7f026c1430c0 2024-03-25T18:28:23.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f0268ff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:23.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f0268ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:23.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f0268ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:23.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f02721c6640 5 monclient: authenticate success, global_id 4352 2024-03-25T18:28:23.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f02721c6640 10 monclient: _renew_subs 2024-03-25T18:28:23.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f02721c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:28:23.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f02721c6640 1 -- 172.21.15.99:0/1137035740 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026c135f70 con 0x7f026c1430c0 2024-03-25T18:28:23.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f02721c6640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:23.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f02721c6640 10 monclient: _renew_subs 2024-03-25T18:28:23.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f02721c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:28:23.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f02721c6640 1 -- 172.21.15.99:0/1137035740 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f026c13b780 con 0x7f026c1430c0 2024-03-25T18:28:23.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.776+0000 7f02721c6640 1 librados: init done 2024-03-25T18:28:23.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.777+0000 7f02721c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:23.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.777+0000 7f02721c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:23.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.777+0000 7f02721c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:28:23.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:23.777+0000 7f02721c6640 1 -- 172.21.15.99:0/1137035740 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f026c0627b0 con 0x7f026c1430c0 2024-03-25T18:28:24.080 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.079+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0264003990 con 0x7f026c1430c0 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.079+0000 7f0268ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.079+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0264003b80 con 0x7f026c1430c0 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.079+0000 7f0268ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.079+0000 7f0268ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.079+0000 7f0268ff9640 10 monclient: dump: 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:24.081 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:24.082 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:24.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.079+0000 7f026a7fc640 4 set_mon_vals no callback set 2024-03-25T18:28:24.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.080+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f0264003da0 con 0x7f026c1430c0 2024-03-25T18:28:24.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.080+0000 7f026b7fe640 10 monclient: get_auth_request con 0x7f024c06e0f0 auth_method 0 2024-03-25T18:28:24.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.081+0000 7f0268ff9640 10 client.4352.objecter ms_handle_connect 0x7f024c06e0f0 2024-03-25T18:28:24.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.086+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f0264054910 con 0x7f026c1430c0 2024-03-25T18:28:24.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.086+0000 7f0268ff9640 10 client.4352.objecter ms_dispatch 0x7f026c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:24.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.086+0000 7f0268ff9640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:24.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.086+0000 7f0268ff9640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:24.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.086+0000 7f0268ff9640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-25T18:28:24.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.086+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0264050420 con 0x7f026c1430c0 2024-03-25T18:28:24.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.086+0000 7f0268ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:24.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.086+0000 7f0268ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:24.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.309+0000 7f02721c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:24.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.309+0000 7f02721c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:24.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.309+0000 7f02721c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:28:24.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.309+0000 7f02721c6640 1 -- 172.21.15.99:0/1137035740 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f026c13e230 con 0x7f026c1430c0 2024-03-25T18:28:24.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.398+0000 7f0268ff9640 1 -- 172.21.15.99:0/1137035740 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f0264052350 con 0x7f026c1430c0 2024-03-25T18:28:24.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.398+0000 7f0268ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:24.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.398+0000 7f0268ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:24.400 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:28:24.400 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:24.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.401+0000 7f02721c6640 10 librados: watch_flush enter 2024-03-25T18:28:24.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.401+0000 7f02721c6640 10 librados: watch_flush exit 2024-03-25T18:28:24.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.401+0000 7f02721c6640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-25T18:28:24.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.401+0000 7f02721c6640 10 client.4352.objecter successfully canceled tick 2024-03-25T18:28:24.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.401+0000 7f02721c6640 1 -- 172.21.15.99:0/1137035740 >> v1:172.21.15.190:6800/4220409699 conn(0x7f024c06e0f0 legacy=0x7f024c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:24.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.401+0000 7f02721c6640 10 monclient: shutdown 2024-03-25T18:28:24.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.401+0000 7f02721c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:24.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.401+0000 7f02721c6640 1 -- 172.21.15.99:0/1137035740 >> v1:172.21.15.99:6790/0 conn(0x7f026c1430c0 legacy=0x7f026c1312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:24.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.401+0000 7f02721c6640 1 -- 172.21.15.99:0/1137035740 shutdown_connections 2024-03-25T18:28:24.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.401+0000 7f02721c6640 1 -- 172.21.15.99:0/1137035740 >> 172.21.15.99:0/1137035740 conn(0x7f026c0b7ea0 msgr2=0x7f026c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:24.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.401+0000 7f02721c6640 1 -- 172.21.15.99:0/1137035740 shutdown_connections 2024-03-25T18:28:24.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.401+0000 7f02721c6640 1 -- 172.21.15.99:0/1137035740 wait complete. 2024-03-25T18:28:24.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:24.402+0000 7f02721c6640 1 librados: shutdown 2024-03-25T18:28:24.415 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:28.960 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:03:29.553 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:28.972 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:03:29.564 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:29.007 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:03:29.599 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:29.009 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:03:29.568 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:29.021 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:03:29.580 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:29.028 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:03:29.621 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:29.042 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:03:29.635 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:29.049 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:03:29.608 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:29.077 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:03:29.670 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:31.223 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:31.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.337+0000 7f2b77996640 10 monclient: get_monmap_and_config 2024-03-25T18:28:31.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.337+0000 7f2b77996640 10 monclient: build_initial_monmap 2024-03-25T18:28:31.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.337+0000 7f2b77996640 10 monclient: monmap: 2024-03-25T18:28:31.338 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:31.339 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:31.339 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:28:31.338141+0000 2024-03-25T18:28:31.339 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:28:31.338141+0000 2024-03-25T18:28:31.339 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:31.339 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:31.339 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:31.339 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:28:31.339 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:28:31.339 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:31.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.337+0000 7f2b77996640 1 Processor -- start 2024-03-25T18:28:31.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.338+0000 7f2b77996640 1 -- start start 2024-03-25T18:28:31.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.338+0000 7f2b77996640 10 monclient: init 2024-03-25T18:28:31.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.339+0000 7f2b77996640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:31.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.339+0000 7f2b77996640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:31.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.339+0000 7f2b77996640 10 monclient(hunting): picked mon.noname-c con 0x7f2b700a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:31.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.339+0000 7f2b77996640 10 monclient(hunting): picked mon.noname-a con 0x7f2b7013f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:31.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.339+0000 7f2b77996640 10 monclient(hunting): picked mon.noname-b con 0x7f2b701430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:31.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.339+0000 7f2b77996640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b70142520 con 0x7f2b7013f710 2024-03-25T18:28:31.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.339+0000 7f2b77996640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b700b28a0 con 0x7f2b700a98f0 2024-03-25T18:28:31.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.339+0000 7f2b77996640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b700b2a30 con 0x7f2b701430c0 2024-03-25T18:28:31.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.339+0000 7f2b77996640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:31.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.339+0000 7f2b77996640 10 monclient(hunting): authenticate will time out at 7529.507220s 2024-03-25T18:28:31.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.350+0000 7f2b75732640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f2b700a98f0 0x7f2b700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:53218/0 (socket says 172.21.15.99:53218) 2024-03-25T18:28:31.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.350+0000 7f2b75732640 1 -- 172.21.15.99:0/1354562257 learned_addr learned my addr 172.21.15.99:0/1354562257 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:31.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.363+0000 7f2b67fff640 1 -- 172.21.15.99:0/1354562257 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868561889 0 0) 0x7f2b700b28a0 con 0x7f2b700a98f0 2024-03-25T18:28:31.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.363+0000 7f2b67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:31.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.363+0000 7f2b67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:31.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.363+0000 7f2b67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:31.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.363+0000 7f2b67fff640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:28:31.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.363+0000 7f2b67fff640 1 -- 172.21.15.99:0/1354562257 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b5c003520 con 0x7f2b700a98f0 2024-03-25T18:28:31.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.364+0000 7f2b67fff640 1 -- 172.21.15.99:0/1354562257 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634721705 0 0) 0x7f2b70142520 con 0x7f2b7013f710 2024-03-25T18:28:31.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.364+0000 7f2b67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:31.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.364+0000 7f2b67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:31.366 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.364+0000 7f2b67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:31.366 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.364+0000 7f2b67fff640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:28:31.366 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.364+0000 7f2b67fff640 1 -- 172.21.15.99:0/1354562257 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b700b28a0 con 0x7f2b7013f710 2024-03-25T18:28:31.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 1 -- 172.21.15.99:0/1354562257 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839388514 0 0) 0x7f2b700b2a30 con 0x7f2b701430c0 2024-03-25T18:28:31.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:31.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:31.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:31.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:28:31.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 1 -- 172.21.15.99:0/1354562257 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b70142520 con 0x7f2b701430c0 2024-03-25T18:28:31.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 1 -- 172.21.15.99:0/1354562257 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 270034532 0 0) 0x7f2b5c003520 con 0x7f2b700a98f0 2024-03-25T18:28:31.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 1 -- 172.21.15.99:0/1354562257 >> v1:172.21.15.99:6790/0 conn(0x7f2b701430c0 legacy=0x7f2b7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:31.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 1 -- 172.21.15.99:0/1354562257 >> v1:172.21.15.99:6789/0 conn(0x7f2b7013f710 legacy=0x7f2b701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:31.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:31.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 1 monclient: found mon.noname-c 2024-03-25T18:28:31.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:31.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:28:31.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 1 -- 172.21.15.99:0/1354562257 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b700b44c0 con 0x7f2b700a98f0 2024-03-25T18:28:31.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 10 monclient: _finish_auth 0 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 1 -- 172.21.15.99:0/1354562257 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2b6c001690 con 0x7f2b700a98f0 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b67fff640 10 monclient: dump: 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:31.373 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:31.374 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:31.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b77996640 5 monclient: authenticate success, global_id 4363 2024-03-25T18:28:31.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.370+0000 7f2b77996640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:31.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.399+0000 7f2b67fff640 1 -- 172.21.15.99:0/1354562257 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b6c0017f0 con 0x7f2b700a98f0 2024-03-25T18:28:31.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.399+0000 7f2b67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:31.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.399+0000 7f2b67fff640 1 -- 172.21.15.99:0/1354562257 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2b6c001c00 con 0x7f2b700a98f0 2024-03-25T18:28:31.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.399+0000 7f2b67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:31.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.399+0000 7f2b67fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:31.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.399+0000 7f2b67fff640 10 monclient: dump: 2024-03-25T18:28:31.401 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:31.401 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:31.401 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:31.401 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:31.401 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:31.401 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:31.401 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:31.402 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:31.402 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:31.402 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:31.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.399+0000 7f2b77996640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:31.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.399+0000 7f2b77996640 4 set_mon_vals no callback set 2024-03-25T18:28:31.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.400+0000 7f2b77996640 10 monclient: shutdown 2024-03-25T18:28:31.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.400+0000 7f2b77996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:31.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.400+0000 7f2b77996640 1 -- 172.21.15.99:0/1354562257 >> v1:172.21.15.190:6789/0 conn(0x7f2b700a98f0 legacy=0x7f2b700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:31.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.400+0000 7f2b77996640 1 -- 172.21.15.99:0/1354562257 shutdown_connections 2024-03-25T18:28:31.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.400+0000 7f2b77996640 1 -- 172.21.15.99:0/1354562257 >> 172.21.15.99:0/1354562257 conn(0x7f2b700b7ea0 msgr2=0x7f2b700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:31.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.400+0000 7f2b77996640 1 -- 172.21.15.99:0/1354562257 shutdown_connections 2024-03-25T18:28:31.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.400+0000 7f2b77996640 1 -- 172.21.15.99:0/1354562257 wait complete. 2024-03-25T18:28:31.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.401+0000 7f2b77996640 10 monclient: build_initial_monmap 2024-03-25T18:28:31.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.401+0000 7f2b77996640 10 monclient: monmap: 2024-03-25T18:28:31.405 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:31.405 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:31.405 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:28:31.405 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:28:31.405 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:31.405 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:28:31.405 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:31.405 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:28:31.405 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:28:31.406 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:31.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.401+0000 7f2b77996640 1 librados: starting msgr at 2024-03-25T18:28:31.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.401+0000 7f2b77996640 1 librados: starting objecter 2024-03-25T18:28:31.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 1 Processor -- start 2024-03-25T18:28:31.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 1 -- start start 2024-03-25T18:28:31.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 1 librados: setting wanted keys 2024-03-25T18:28:31.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 1 librados: calling monclient init 2024-03-25T18:28:31.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 10 monclient: init 2024-03-25T18:28:31.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:31.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:31.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 10 monclient(hunting): picked mon.noname-a con 0x7f2b700a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:31.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 10 monclient(hunting): picked mon.noname-c con 0x7f2b7013f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:31.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 10 monclient(hunting): picked mon.noname-b con 0x7f2b701430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:31.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b70138410 con 0x7f2b700a98f0 2024-03-25T18:28:31.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b70138120 con 0x7f2b701430c0 2024-03-25T18:28:31.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b70139490 con 0x7f2b7013f710 2024-03-25T18:28:31.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:31.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.402+0000 7f2b77996640 10 monclient(hunting): authenticate will time out at 7529.570760s 2024-03-25T18:28:31.411 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.410+0000 7f2b75732640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2b700a98f0 0x7f2b70135000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:52502/0 (socket says 172.21.15.99:52502) 2024-03-25T18:28:31.411 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.410+0000 7f2b75732640 1 -- 172.21.15.99:0/2790948287 learned_addr learned my addr 172.21.15.99:0/2790948287 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:31.414 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.414+0000 7f2b65ffb640 10 client.?.objecter ms_handle_connect 0x7f2b700a98f0 2024-03-25T18:28:31.414 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.414+0000 7f2b65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:31.422 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.421+0000 7f2b65ffb640 10 client.?.objecter ms_handle_connect 0x7f2b7013f710 2024-03-25T18:28:31.422 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.421+0000 7f2b65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:31.423 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.422+0000 7f2b65ffb640 10 client.?.objecter ms_handle_connect 0x7f2b701430c0 2024-03-25T18:28:31.423 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.422+0000 7f2b65ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:31.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.423+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221263451 0 0) 0x7f2b70138410 con 0x7f2b700a98f0 2024-03-25T18:28:31.425 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.423+0000 7f2b65ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:31.425 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.424+0000 7f2b65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:31.425 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.424+0000 7f2b65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:31.425 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.424+0000 7f2b65ffb640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:28:31.425 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.424+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b54004eb0 con 0x7f2b700a98f0 2024-03-25T18:28:31.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.432+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547890904 0 0) 0x7f2b70139490 con 0x7f2b7013f710 2024-03-25T18:28:31.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.432+0000 7f2b65ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:31.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.432+0000 7f2b65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:31.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.432+0000 7f2b65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:31.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.432+0000 7f2b65ffb640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:28:31.434 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.432+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b70138410 con 0x7f2b7013f710 2024-03-25T18:28:31.434 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.432+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630644149 0 0) 0x7f2b70138120 con 0x7f2b701430c0 2024-03-25T18:28:31.434 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.432+0000 7f2b65ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:31.434 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.432+0000 7f2b65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:31.434 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.432+0000 7f2b65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:31.434 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.432+0000 7f2b65ffb640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:28:31.434 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.432+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b70139490 con 0x7f2b701430c0 2024-03-25T18:28:31.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.439+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2281724356 0 0) 0x7f2b70139490 con 0x7f2b701430c0 2024-03-25T18:28:31.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.439+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b70138120 con 0x7f2b701430c0 2024-03-25T18:28:31.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.439+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2b68001bd0 con 0x7f2b701430c0 2024-03-25T18:28:31.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.439+0000 7f2b65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:31.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.439+0000 7f2b65ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:31.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.439+0000 7f2b65ffb640 10 monclient(hunting): dump: 2024-03-25T18:28:31.440 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:31.440 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:31.441 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:31.441 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:31.441 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:31.441 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:31.441 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:31.441 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:31.441 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:31.441 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:31.441 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.439+0000 7f2b65ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:31.443 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.442+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1765873443 0 0) 0x7f2b70138410 con 0x7f2b7013f710 2024-03-25T18:28:31.443 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.442+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b70139490 con 0x7f2b7013f710 2024-03-25T18:28:31.443 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.442+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2b60001970 con 0x7f2b7013f710 2024-03-25T18:28:31.443 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.442+0000 7f2b65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:31.443 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.442+0000 7f2b65ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:31.443 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.442+0000 7f2b65ffb640 10 monclient(hunting): dump: 2024-03-25T18:28:31.443 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:31.443 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:31.443 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:31.443 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:31.443 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:31.443 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:31.444 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:31.444 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:31.444 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:31.444 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:31.445 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.444+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3702776631 0 0) 0x7f2b54004eb0 con 0x7f2b700a98f0 2024-03-25T18:28:31.445 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.444+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b70138410 con 0x7f2b700a98f0 2024-03-25T18:28:31.445 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.444+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2b6c001b40 con 0x7f2b700a98f0 2024-03-25T18:28:31.445 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.444+0000 7f2b65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:31.445 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.444+0000 7f2b65ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:31.445 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.444+0000 7f2b65ffb640 10 monclient(hunting): dump: 2024-03-25T18:28:31.445 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:31.445 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:31.445 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:31.446 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:31.446 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:31.446 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:31.446 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:31.446 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:31.446 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:31.446 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:31.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 171664254 0 0) 0x7f2b70138410 con 0x7f2b700a98f0 2024-03-25T18:28:31.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 >> v1:172.21.15.99:6790/0 conn(0x7f2b7013f710 legacy=0x7f2b70132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:31.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 >> v1:172.21.15.190:6789/0 conn(0x7f2b701430c0 legacy=0x7f2b701332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:31.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b65ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:28:31.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b65ffb640 1 monclient: found mon.a 2024-03-25T18:28:31.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:31.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b65ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:31.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b70139770 con 0x7f2b700a98f0 2024-03-25T18:28:31.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b65ffb640 10 monclient: _finish_auth 0 2024-03-25T18:28:31.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b65ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:28:31.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:31.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b77996640 5 monclient: authenticate success, global_id 4350 2024-03-25T18:28:31.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b77996640 10 monclient: _renew_subs 2024-03-25T18:28:31.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b77996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:31.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.452+0000 7f2b77996640 1 -- 172.21.15.99:0/2790948287 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b701339d0 con 0x7f2b700a98f0 2024-03-25T18:28:31.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.453+0000 7f2b77996640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:31.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.453+0000 7f2b77996640 10 monclient: _renew_subs 2024-03-25T18:28:31.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.453+0000 7f2b77996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:31.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.453+0000 7f2b77996640 1 -- 172.21.15.99:0/2790948287 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b700b0d80 con 0x7f2b700a98f0 2024-03-25T18:28:31.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.453+0000 7f2b77996640 1 librados: init done 2024-03-25T18:28:31.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.453+0000 7f2b77996640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:31.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.453+0000 7f2b77996640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:31.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.453+0000 7f2b77996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:31.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.453+0000 7f2b77996640 1 -- 172.21.15.99:0/2790948287 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b700b1080 con 0x7f2b700a98f0 2024-03-25T18:28:31.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.762+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b6c004a90 con 0x7f2b700a98f0 2024-03-25T18:28:31.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.762+0000 7f2b65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:31.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.762+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2b6c004c00 con 0x7f2b700a98f0 2024-03-25T18:28:31.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.762+0000 7f2b65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:31.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.762+0000 7f2b65ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:31.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.762+0000 7f2b65ffb640 10 monclient: dump: 2024-03-25T18:28:31.763 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:31.763 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:31.763 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:31.764 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:31.764 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:31.764 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:31.764 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:31.764 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:31.764 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:31.764 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:31.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.762+0000 7f2b677fe640 4 set_mon_vals no callback set 2024-03-25T18:28:31.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.762+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f2b6c01abd0 con 0x7f2b700a98f0 2024-03-25T18:28:31.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.763+0000 7f2b74f31640 10 monclient: get_auth_request con 0x7f2b5406e100 auth_method 0 2024-03-25T18:28:31.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.763+0000 7f2b65ffb640 10 client.4350.objecter ms_handle_connect 0x7f2b5406e100 2024-03-25T18:28:31.770 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.769+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f2b6c054790 con 0x7f2b700a98f0 2024-03-25T18:28:31.770 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.769+0000 7f2b65ffb640 10 client.4350.objecter ms_dispatch 0x7f2b70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:31.771 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.769+0000 7f2b65ffb640 3 client.4350.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:31.771 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.769+0000 7f2b65ffb640 3 client.4350.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:31.771 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.769+0000 7f2b65ffb640 20 client.4350.objecter dump_active .. 0 homeless 2024-03-25T18:28:31.771 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.769+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b6c0502a0 con 0x7f2b700a98f0 2024-03-25T18:28:31.771 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.769+0000 7f2b65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:31.771 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:31.769+0000 7f2b65ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:32.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.021+0000 7f2b77996640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:32.023 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.021+0000 7f2b77996640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:32.023 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.021+0000 7f2b77996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:28:32.023 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.021+0000 7f2b77996640 1 -- 172.21.15.99:0/2790948287 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b700627b0 con 0x7f2b700a98f0 2024-03-25T18:28:32.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.063+0000 7f2b65ffb640 1 -- 172.21.15.99:0/2790948287 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f2b6c0521d0 con 0x7f2b700a98f0 2024-03-25T18:28:32.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.063+0000 7f2b65ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:32.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.063+0000 7f2b65ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:32.065 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:28:32.065 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:32.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.066+0000 7f2b77996640 10 librados: watch_flush enter 2024-03-25T18:28:32.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.066+0000 7f2b77996640 10 librados: watch_flush exit 2024-03-25T18:28:32.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.066+0000 7f2b77996640 20 client.4350.objecter shutdown clearing up homeless session... 2024-03-25T18:28:32.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.066+0000 7f2b77996640 10 client.4350.objecter successfully canceled tick 2024-03-25T18:28:32.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.066+0000 7f2b77996640 1 -- 172.21.15.99:0/2790948287 >> v1:172.21.15.190:6800/4220409699 conn(0x7f2b5406e100 legacy=0x7f2b540705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:32.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.066+0000 7f2b77996640 10 monclient: shutdown 2024-03-25T18:28:32.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.066+0000 7f2b77996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:32.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.066+0000 7f2b77996640 1 -- 172.21.15.99:0/2790948287 >> v1:172.21.15.99:6789/0 conn(0x7f2b700a98f0 legacy=0x7f2b70135000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:32.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.066+0000 7f2b77996640 1 -- 172.21.15.99:0/2790948287 shutdown_connections 2024-03-25T18:28:32.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.066+0000 7f2b77996640 1 -- 172.21.15.99:0/2790948287 >> 172.21.15.99:0/2790948287 conn(0x7f2b700b7ea0 msgr2=0x7f2b70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:32.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.066+0000 7f2b77996640 1 -- 172.21.15.99:0/2790948287 shutdown_connections 2024-03-25T18:28:32.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.066+0000 7f2b77996640 1 -- 172.21.15.99:0/2790948287 wait complete. 2024-03-25T18:28:32.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:32.066+0000 7f2b77996640 1 librados: shutdown 2024-03-25T18:28:32.079 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:38.886 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:39.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.000+0000 7f70976c1640 10 monclient: get_monmap_and_config 2024-03-25T18:28:39.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.000+0000 7f70976c1640 10 monclient: build_initial_monmap 2024-03-25T18:28:39.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.000+0000 7f70976c1640 10 monclient: monmap: 2024-03-25T18:28:39.001 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:39.001 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:39.001 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:28:39.001291+0000 2024-03-25T18:28:39.001 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:28:39.001291+0000 2024-03-25T18:28:39.002 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:39.002 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:39.002 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:39.002 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:28:39.002 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:28:39.002 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:39.003 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.000+0000 7f70976c1640 1 Processor -- start 2024-03-25T18:28:39.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.001+0000 7f70976c1640 1 -- start start 2024-03-25T18:28:39.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.001+0000 7f70976c1640 10 monclient: init 2024-03-25T18:28:39.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.001+0000 7f70976c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:39.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.001+0000 7f70976c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:39.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.001+0000 7f70976c1640 10 monclient(hunting): picked mon.noname-a con 0x7f70900a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:39.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.001+0000 7f70976c1640 10 monclient(hunting): picked mon.noname-b con 0x7f709013f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:39.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.001+0000 7f70976c1640 10 monclient(hunting): picked mon.noname-c con 0x7f70901430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:39.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.001+0000 7f70976c1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7090142520 con 0x7f70900a98f0 2024-03-25T18:28:39.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.001+0000 7f70976c1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70900b28a0 con 0x7f70901430c0 2024-03-25T18:28:39.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.001+0000 7f70976c1640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70900b2a30 con 0x7f709013f710 2024-03-25T18:28:39.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.001+0000 7f70976c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:39.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.001+0000 7f70976c1640 10 monclient(hunting): authenticate will time out at 7537.169581s 2024-03-25T18:28:39.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.002+0000 7f7094c5c640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f709013f710 0x7f70901429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:51594/0 (socket says 172.21.15.99:51594) 2024-03-25T18:28:39.005 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.002+0000 7f7094c5c640 1 -- 172.21.15.99:0/912589219 learned_addr learned my addr 172.21.15.99:0/912589219 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:39.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.068+0000 7f7087fff640 1 -- 172.21.15.99:0/912589219 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666166363 0 0) 0x7f70900b28a0 con 0x7f70901430c0 2024-03-25T18:28:39.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.069+0000 7f7087fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:39.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.069+0000 7f7087fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:39.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.069+0000 7f7087fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:39.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.069+0000 7f7087fff640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:28:39.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.069+0000 7f7087fff640 1 -- 172.21.15.99:0/912589219 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f707c003520 con 0x7f70901430c0 2024-03-25T18:28:39.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.077+0000 7f7087fff640 1 -- 172.21.15.99:0/912589219 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712645083 0 0) 0x7f70900b2a30 con 0x7f709013f710 2024-03-25T18:28:39.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.077+0000 7f7087fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:39.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.077+0000 7f7087fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:39.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.077+0000 7f7087fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:39.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.077+0000 7f7087fff640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:28:39.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.077+0000 7f7087fff640 1 -- 172.21.15.99:0/912589219 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70900b28a0 con 0x7f709013f710 2024-03-25T18:28:39.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f7087fff640 1 -- 172.21.15.99:0/912589219 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2231279059 0 0) 0x7f707c003520 con 0x7f70901430c0 2024-03-25T18:28:39.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f7087fff640 1 -- 172.21.15.99:0/912589219 >> v1:172.21.15.99:6790/0 conn(0x7f709013f710 legacy=0x7f70901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:39.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f7087fff640 1 -- 172.21.15.99:0/912589219 >> v1:172.21.15.99:6789/0 conn(0x7f70900a98f0 legacy=0x7f70900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:39.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f7087fff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:39.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f7087fff640 1 monclient: found mon.noname-c 2024-03-25T18:28:39.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f7087fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:39.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f7087fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:28:39.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f7087fff640 1 -- 172.21.15.99:0/912589219 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70900b44c0 con 0x7f70901430c0 2024-03-25T18:28:39.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f7087fff640 10 monclient: _finish_auth 0 2024-03-25T18:28:39.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f7087fff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:39.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f7087fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:39.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f7087fff640 1 -- 172.21.15.99:0/912589219 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f7088001a10 con 0x7f70901430c0 2024-03-25T18:28:39.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f70976c1640 5 monclient: authenticate success, global_id 4369 2024-03-25T18:28:39.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f70976c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:39.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.091+0000 7f7087fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:39.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.092+0000 7f7087fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:39.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.092+0000 7f7087fff640 10 monclient: dump: 2024-03-25T18:28:39.094 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:39.094 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:39.094 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:39.094 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:39.094 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:39.094 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:39.094 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:39.095 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:39.095 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:39.095 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:39.095 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.092+0000 7f70976c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:39.098 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.097+0000 7f7087fff640 1 -- 172.21.15.99:0/912589219 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7088001b70 con 0x7f70901430c0 2024-03-25T18:28:39.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.097+0000 7f7087fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:39.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.097+0000 7f7087fff640 1 -- 172.21.15.99:0/912589219 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f7088001f80 con 0x7f70901430c0 2024-03-25T18:28:39.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.097+0000 7f70976c1640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:39.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.097+0000 7f70976c1640 4 set_mon_vals no callback set 2024-03-25T18:28:39.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.098+0000 7f7087fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:39.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.098+0000 7f7087fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:39.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.098+0000 7f7087fff640 10 monclient: dump: 2024-03-25T18:28:39.099 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:39.099 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:39.099 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:39.099 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:39.099 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:39.099 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:39.099 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:39.100 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:39.100 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:39.100 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:39.102 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.098+0000 7f70976c1640 10 monclient: shutdown 2024-03-25T18:28:39.102 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.098+0000 7f70976c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:39.103 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.098+0000 7f70976c1640 1 -- 172.21.15.99:0/912589219 >> v1:172.21.15.190:6789/0 conn(0x7f70901430c0 legacy=0x7f709014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:39.103 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.098+0000 7f70976c1640 1 -- 172.21.15.99:0/912589219 shutdown_connections 2024-03-25T18:28:39.103 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.098+0000 7f70976c1640 1 -- 172.21.15.99:0/912589219 >> 172.21.15.99:0/912589219 conn(0x7f70900b7ea0 msgr2=0x7f70900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:39.103 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.099+0000 7f70976c1640 1 -- 172.21.15.99:0/912589219 shutdown_connections 2024-03-25T18:28:39.103 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.099+0000 7f70976c1640 1 -- 172.21.15.99:0/912589219 wait complete. 2024-03-25T18:28:39.103 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.099+0000 7f70976c1640 10 monclient: build_initial_monmap 2024-03-25T18:28:39.103 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.099+0000 7f70976c1640 10 monclient: monmap: 2024-03-25T18:28:39.103 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:39.103 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:39.103 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:28:39.103 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:28:39.103 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:39.104 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:28:39.104 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:39.104 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:28:39.104 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:28:39.104 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:39.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.100+0000 7f70976c1640 1 librados: starting msgr at 2024-03-25T18:28:39.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.100+0000 7f70976c1640 1 librados: starting objecter 2024-03-25T18:28:39.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.100+0000 7f70976c1640 1 Processor -- start 2024-03-25T18:28:39.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.100+0000 7f70976c1640 1 -- start start 2024-03-25T18:28:39.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.100+0000 7f70976c1640 1 librados: setting wanted keys 2024-03-25T18:28:39.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.100+0000 7f70976c1640 1 librados: calling monclient init 2024-03-25T18:28:39.105 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.100+0000 7f70976c1640 10 monclient: init 2024-03-25T18:28:39.105 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.101+0000 7f70976c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:39.105 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.101+0000 7f70976c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:39.105 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.101+0000 7f70976c1640 10 monclient(hunting): picked mon.noname-b con 0x7f70900a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:39.105 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.101+0000 7f70976c1640 10 monclient(hunting): picked mon.noname-a con 0x7f709013f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:39.105 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.101+0000 7f70976c1640 10 monclient(hunting): picked mon.noname-c con 0x7f70901430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:39.105 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.101+0000 7f70976c1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7090070520 con 0x7f709013f710 2024-03-25T18:28:39.105 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.101+0000 7f70976c1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70901dee90 con 0x7f70900a98f0 2024-03-25T18:28:39.105 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.101+0000 7f70976c1640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709006dee0 con 0x7f70901430c0 2024-03-25T18:28:39.105 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.101+0000 7f70976c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:39.105 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.101+0000 7f70976c1640 10 monclient(hunting): authenticate will time out at 7537.269341s 2024-03-25T18:28:39.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.107+0000 7f709545d640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f70900a98f0 0x7f70901df3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:38894/0 (socket says 172.21.15.99:38894) 2024-03-25T18:28:39.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.107+0000 7f709545d640 1 -- 172.21.15.99:0/211749225 learned_addr learned my addr 172.21.15.99:0/211749225 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:39.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.108+0000 7f7085ffb640 10 client.?.objecter ms_handle_connect 0x7f70900a98f0 2024-03-25T18:28:39.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.108+0000 7f7085ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:39.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.116+0000 7f7085ffb640 10 client.?.objecter ms_handle_connect 0x7f70901430c0 2024-03-25T18:28:39.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.116+0000 7f7085ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:39.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.118+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857113302 0 0) 0x7f70901dee90 con 0x7f70900a98f0 2024-03-25T18:28:39.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.118+0000 7f7085ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:39.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.118+0000 7f7085ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:39.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.118+0000 7f7085ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:39.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.118+0000 7f7085ffb640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:28:39.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.118+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7074004eb0 con 0x7f70900a98f0 2024-03-25T18:28:39.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.121+0000 7f7085ffb640 10 client.?.objecter ms_handle_connect 0x7f709013f710 2024-03-25T18:28:39.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.121+0000 7f7085ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:39.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.128+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683880861 0 0) 0x7f709006dee0 con 0x7f70901430c0 2024-03-25T18:28:39.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.128+0000 7f7085ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:39.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.128+0000 7f7085ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:39.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.128+0000 7f7085ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:39.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.128+0000 7f7085ffb640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:28:39.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.128+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70901dee90 con 0x7f70901430c0 2024-03-25T18:28:39.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.133+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1109288661 0 0) 0x7f7074004eb0 con 0x7f70900a98f0 2024-03-25T18:28:39.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.133+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f709006dee0 con 0x7f70900a98f0 2024-03-25T18:28:39.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.133+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f708c001a30 con 0x7f70900a98f0 2024-03-25T18:28:39.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.133+0000 7f7085ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:39.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.133+0000 7f7085ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:39.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.133+0000 7f7085ffb640 10 monclient(hunting): dump: 2024-03-25T18:28:39.135 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:39.135 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:39.135 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:39.135 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:39.135 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:39.135 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:39.135 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:39.135 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:39.135 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:39.135 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:39.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.133+0000 7f7085ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:39.149 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2797587659 0 0) 0x7f709006dee0 con 0x7f70900a98f0 2024-03-25T18:28:39.149 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 >> v1:172.21.15.99:6790/0 conn(0x7f70901430c0 legacy=0x7f7090138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:39.149 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 >> v1:172.21.15.99:6789/0 conn(0x7f709013f710 legacy=0x7f709006e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:39.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f7085ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:28:39.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f7085ffb640 1 monclient: found mon.b 2024-03-25T18:28:39.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f7085ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:39.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f7085ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:39.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7090139d20 con 0x7f70900a98f0 2024-03-25T18:28:39.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f7085ffb640 10 monclient: _finish_auth 0 2024-03-25T18:28:39.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f7085ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:28:39.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f7085ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:39.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f70976c1640 5 monclient: authenticate success, global_id 4372 2024-03-25T18:28:39.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f70976c1640 10 monclient: _renew_subs 2024-03-25T18:28:39.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f70976c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:39.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f70976c1640 1 -- 172.21.15.99:0/211749225 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70901dfac0 con 0x7f70900a98f0 2024-03-25T18:28:39.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f70976c1640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:39.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f70976c1640 10 monclient: _renew_subs 2024-03-25T18:28:39.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f70976c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:39.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f70976c1640 1 -- 172.21.15.99:0/211749225 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f709006f2b0 con 0x7f70900a98f0 2024-03-25T18:28:39.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.148+0000 7f70976c1640 1 librados: init done 2024-03-25T18:28:39.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.149+0000 7f70976c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:39.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.149+0000 7f70976c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:39.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.149+0000 7f70976c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:39.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.149+0000 7f70976c1640 1 -- 172.21.15.99:0/211749225 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70900627b0 con 0x7f70900a98f0 2024-03-25T18:28:39.158 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.157+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f708c001c70 con 0x7f70900a98f0 2024-03-25T18:28:39.158 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.157+0000 7f7085ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:39.158 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.157+0000 7f70877fe640 4 set_mon_vals no callback set 2024-03-25T18:28:39.420 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.419+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f708c003a60 con 0x7f70900a98f0 2024-03-25T18:28:39.420 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.419+0000 7f7085ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:39.420 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.419+0000 7f7085ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:39.420 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.419+0000 7f7085ffb640 10 monclient: dump: 2024-03-25T18:28:39.420 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:39.420 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:39.421 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:39.421 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:39.421 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:39.421 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:39.421 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:39.421 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:39.421 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:39.421 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:39.421 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.421+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f708c003c40 con 0x7f70900a98f0 2024-03-25T18:28:39.422 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.421+0000 7f7094c5c640 10 monclient: get_auth_request con 0x7f707406dd50 auth_method 0 2024-03-25T18:28:39.422 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.422+0000 7f7085ffb640 10 client.4372.objecter ms_handle_connect 0x7f707406dd50 2024-03-25T18:28:39.425 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.424+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f708c054940 con 0x7f70900a98f0 2024-03-25T18:28:39.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.424+0000 7f7085ffb640 10 client.4372.objecter ms_dispatch 0x7f7090000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:39.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.424+0000 7f7085ffb640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:39.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.424+0000 7f7085ffb640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:39.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.424+0000 7f7085ffb640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T18:28:39.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.428+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f708c050450 con 0x7f70900a98f0 2024-03-25T18:28:39.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.428+0000 7f7085ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:39.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.428+0000 7f7085ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:39.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.679+0000 7f70976c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:39.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.679+0000 7f70976c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:39.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.679+0000 7f70976c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:39.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.679+0000 7f70976c1640 1 -- 172.21.15.99:0/211749225 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7090069f40 con 0x7f70900a98f0 2024-03-25T18:28:39.704 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.703+0000 7f7085ffb640 1 -- 172.21.15.99:0/211749225 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f708c052380 con 0x7f70900a98f0 2024-03-25T18:28:39.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.703+0000 7f7085ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:39.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.703+0000 7f7085ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:39.705 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:28:39.705 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:39.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.706+0000 7f70976c1640 10 librados: watch_flush enter 2024-03-25T18:28:39.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.706+0000 7f70976c1640 10 librados: watch_flush exit 2024-03-25T18:28:39.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.706+0000 7f70976c1640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T18:28:39.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.706+0000 7f70976c1640 10 client.4372.objecter successfully canceled tick 2024-03-25T18:28:39.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.706+0000 7f70976c1640 1 -- 172.21.15.99:0/211749225 >> v1:172.21.15.190:6800/4220409699 conn(0x7f707406dd50 legacy=0x7f7074070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:39.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.706+0000 7f70976c1640 10 monclient: shutdown 2024-03-25T18:28:39.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.706+0000 7f70976c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:39.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.706+0000 7f70976c1640 1 -- 172.21.15.99:0/211749225 >> v1:172.21.15.190:6789/0 conn(0x7f70900a98f0 legacy=0x7f70901df3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:39.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.706+0000 7f70976c1640 1 -- 172.21.15.99:0/211749225 shutdown_connections 2024-03-25T18:28:39.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.706+0000 7f70976c1640 1 -- 172.21.15.99:0/211749225 >> 172.21.15.99:0/211749225 conn(0x7f70900b7ea0 msgr2=0x7f709013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:39.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.706+0000 7f70976c1640 1 -- 172.21.15.99:0/211749225 shutdown_connections 2024-03-25T18:28:39.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.706+0000 7f70976c1640 1 -- 172.21.15.99:0/211749225 wait complete. 2024-03-25T18:28:39.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:39.706+0000 7f70976c1640 1 librados: shutdown 2024-03-25T18:28:39.719 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:46.524 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:46.641 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.640+0000 7f5f3caf1640 10 monclient: get_monmap_and_config 2024-03-25T18:28:46.641 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.640+0000 7f5f3caf1640 10 monclient: build_initial_monmap 2024-03-25T18:28:46.641 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.640+0000 7f5f3caf1640 10 monclient: monmap: 2024-03-25T18:28:46.641 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:46.641 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:46.642 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:28:46.641242+0000 2024-03-25T18:28:46.642 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:28:46.641242+0000 2024-03-25T18:28:46.642 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:46.642 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:46.642 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:46.642 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:28:46.642 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:28:46.642 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:46.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.640+0000 7f5f3caf1640 1 Processor -- start 2024-03-25T18:28:46.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.640+0000 7f5f3caf1640 1 -- start start 2024-03-25T18:28:46.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.640+0000 7f5f3caf1640 10 monclient: init 2024-03-25T18:28:46.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.641+0000 7f5f3caf1640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:46.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.641+0000 7f5f3caf1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:46.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.641+0000 7f5f3caf1640 10 monclient(hunting): picked mon.noname-b con 0x7f5f380a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:46.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.641+0000 7f5f3caf1640 10 monclient(hunting): picked mon.noname-c con 0x7f5f3813f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:46.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.641+0000 7f5f3caf1640 10 monclient(hunting): picked mon.noname-a con 0x7f5f381430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:46.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.641+0000 7f5f3caf1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f38142520 con 0x7f5f381430c0 2024-03-25T18:28:46.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.642+0000 7f5f3caf1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f380b28a0 con 0x7f5f3813f710 2024-03-25T18:28:46.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.642+0000 7f5f3caf1640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f380b2a30 con 0x7f5f380a98f0 2024-03-25T18:28:46.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.642+0000 7f5f3caf1640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:46.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.642+0000 7f5f3caf1640 10 monclient(hunting): authenticate will time out at 7544.809982s 2024-03-25T18:28:46.652 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.651+0000 7f5f35d9b640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f5f3813f710 0x7f5f381429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:45526/0 (socket says 172.21.15.99:45526) 2024-03-25T18:28:46.652 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.651+0000 7f5f35d9b640 1 -- 172.21.15.99:0/2093508539 learned_addr learned my addr 172.21.15.99:0/2093508539 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:46.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.667+0000 7f5f3559a640 1 -- 172.21.15.99:0/2093508539 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726534211 0 0) 0x7f5f380b28a0 con 0x7f5f3813f710 2024-03-25T18:28:46.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.667+0000 7f5f3559a640 10 monclient(hunting): _negotiate 2024-03-25T18:28:46.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.667+0000 7f5f3559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:46.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.667+0000 7f5f3559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:46.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.667+0000 7f5f3559a640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:28:46.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.667+0000 7f5f3559a640 1 -- 172.21.15.99:0/2093508539 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f20003520 con 0x7f5f3813f710 2024-03-25T18:28:46.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.675+0000 7f5f3559a640 1 -- 172.21.15.99:0/2093508539 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 841731075 0 0) 0x7f5f20003520 con 0x7f5f3813f710 2024-03-25T18:28:46.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.675+0000 7f5f3559a640 1 -- 172.21.15.99:0/2093508539 >> v1:172.21.15.99:6790/0 conn(0x7f5f380a98f0 legacy=0x7f5f380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:46.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.675+0000 7f5f3559a640 1 -- 172.21.15.99:0/2093508539 >> v1:172.21.15.99:6789/0 conn(0x7f5f381430c0 legacy=0x7f5f3814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:46.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.675+0000 7f5f3559a640 10 monclient: _finish_hunting 0 2024-03-25T18:28:46.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.675+0000 7f5f3559a640 1 monclient: found mon.noname-c 2024-03-25T18:28:46.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.675+0000 7f5f3559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:46.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.675+0000 7f5f3559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:28:46.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.675+0000 7f5f3559a640 1 -- 172.21.15.99:0/2093508539 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f380b44c0 con 0x7f5f3813f710 2024-03-25T18:28:46.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.675+0000 7f5f3559a640 10 monclient: _finish_auth 0 2024-03-25T18:28:46.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.675+0000 7f5f3559a640 10 monclient: _check_auth_tickets 2024-03-25T18:28:46.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.675+0000 7f5f3559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:46.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.675+0000 7f5f3caf1640 5 monclient: authenticate success, global_id 4375 2024-03-25T18:28:46.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.675+0000 7f5f3caf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:46.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.680+0000 7f5f3559a640 1 -- 172.21.15.99:0/2093508539 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5f2c001690 con 0x7f5f3813f710 2024-03-25T18:28:46.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.680+0000 7f5f3559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:46.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.680+0000 7f5f3559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:46.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.680+0000 7f5f3559a640 10 monclient: dump: 2024-03-25T18:28:46.681 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:46.681 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:46.681 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:46.681 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:46.681 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:46.681 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:46.682 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:46.682 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:46.682 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:46.682 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:46.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.680+0000 7f5f3caf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:46.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.693+0000 7f5f3559a640 1 -- 172.21.15.99:0/2093508539 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f2c001920 con 0x7f5f3813f710 2024-03-25T18:28:46.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.693+0000 7f5f3559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:46.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.693+0000 7f5f3559a640 1 -- 172.21.15.99:0/2093508539 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5f2c001c30 con 0x7f5f3813f710 2024-03-25T18:28:46.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.693+0000 7f5f3559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:46.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.693+0000 7f5f3559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:46.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.693+0000 7f5f3559a640 10 monclient: dump: 2024-03-25T18:28:46.695 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:46.695 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:46.695 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:46.695 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:46.695 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:46.695 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:46.695 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:46.695 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:46.695 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:46.695 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:46.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.693+0000 7f5f3caf1640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:46.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.693+0000 7f5f3caf1640 4 set_mon_vals no callback set 2024-03-25T18:28:46.698 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.694+0000 7f5f3caf1640 10 monclient: shutdown 2024-03-25T18:28:46.698 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.694+0000 7f5f3caf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:46.698 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.694+0000 7f5f3caf1640 1 -- 172.21.15.99:0/2093508539 >> v1:172.21.15.190:6789/0 conn(0x7f5f3813f710 legacy=0x7f5f381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:46.698 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.694+0000 7f5f3caf1640 1 -- 172.21.15.99:0/2093508539 shutdown_connections 2024-03-25T18:28:46.698 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.694+0000 7f5f3caf1640 1 -- 172.21.15.99:0/2093508539 >> 172.21.15.99:0/2093508539 conn(0x7f5f380b7ea0 msgr2=0x7f5f380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:46.698 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.694+0000 7f5f3caf1640 1 -- 172.21.15.99:0/2093508539 shutdown_connections 2024-03-25T18:28:46.698 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.694+0000 7f5f3caf1640 1 -- 172.21.15.99:0/2093508539 wait complete. 2024-03-25T18:28:46.698 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.695+0000 7f5f3caf1640 10 monclient: build_initial_monmap 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.695+0000 7f5f3caf1640 10 monclient: monmap: 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.695+0000 7f5f3caf1640 1 librados: starting msgr at 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.695+0000 7f5f3caf1640 1 librados: starting objecter 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.695+0000 7f5f3caf1640 1 Processor -- start 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.695+0000 7f5f3caf1640 1 -- start start 2024-03-25T18:28:46.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.695+0000 7f5f3caf1640 1 librados: setting wanted keys 2024-03-25T18:28:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.695+0000 7f5f3caf1640 1 librados: calling monclient init 2024-03-25T18:28:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.695+0000 7f5f3caf1640 10 monclient: init 2024-03-25T18:28:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.696+0000 7f5f3caf1640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.696+0000 7f5f3caf1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.696+0000 7f5f3caf1640 10 monclient(hunting): picked mon.noname-c con 0x7f5f380a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.696+0000 7f5f3caf1640 10 monclient(hunting): picked mon.noname-a con 0x7f5f3813f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.696+0000 7f5f3caf1640 10 monclient(hunting): picked mon.noname-b con 0x7f5f381430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.696+0000 7f5f3caf1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f38138d10 con 0x7f5f3813f710 2024-03-25T18:28:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.696+0000 7f5f3caf1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f381d7730 con 0x7f5f381430c0 2024-03-25T18:28:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.696+0000 7f5f3caf1640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f38139900 con 0x7f5f380a98f0 2024-03-25T18:28:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.696+0000 7f5f3caf1640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:46.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.696+0000 7f5f3caf1640 10 monclient(hunting): authenticate will time out at 7544.864185s 2024-03-25T18:28:46.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.703+0000 7f5f36d9d640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f5f381430c0 0x7f5f381d47f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:45570/0 (socket says 172.21.15.99:45570) 2024-03-25T18:28:46.704 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.703+0000 7f5f36d9d640 1 -- 172.21.15.99:0/1851844553 learned_addr learned my addr 172.21.15.99:0/1851844553 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:46.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.705+0000 7f5f26ffd640 10 client.?.objecter ms_handle_connect 0x7f5f381430c0 2024-03-25T18:28:46.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.705+0000 7f5f26ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:46.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.707+0000 7f5f26ffd640 10 client.?.objecter ms_handle_connect 0x7f5f380a98f0 2024-03-25T18:28:46.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.707+0000 7f5f26ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:46.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.710+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793975897 0 0) 0x7f5f381d7730 con 0x7f5f381430c0 2024-03-25T18:28:46.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.711+0000 7f5f26ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:46.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.711+0000 7f5f26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:46.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.711+0000 7f5f26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:46.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.711+0000 7f5f26ffd640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:28:46.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.711+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f18004eb0 con 0x7f5f381430c0 2024-03-25T18:28:46.716 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.716+0000 7f5f26ffd640 10 client.?.objecter ms_handle_connect 0x7f5f3813f710 2024-03-25T18:28:46.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.716+0000 7f5f26ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:46.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.717+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668326033 0 0) 0x7f5f38139900 con 0x7f5f380a98f0 2024-03-25T18:28:46.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.717+0000 7f5f26ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:46.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.717+0000 7f5f26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:46.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.717+0000 7f5f26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:46.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.717+0000 7f5f26ffd640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:28:46.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.717+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f381d7730 con 0x7f5f380a98f0 2024-03-25T18:28:46.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.730+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1692167893 0 0) 0x7f5f381d7730 con 0x7f5f380a98f0 2024-03-25T18:28:46.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.730+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f38139900 con 0x7f5f380a98f0 2024-03-25T18:28:46.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.730+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5f280018a0 con 0x7f5f380a98f0 2024-03-25T18:28:46.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.731+0000 7f5f26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:46.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.731+0000 7f5f26ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:46.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.731+0000 7f5f26ffd640 10 monclient(hunting): dump: 2024-03-25T18:28:46.732 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:46.732 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:46.732 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:46.732 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:46.732 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:46.732 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:46.732 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:46.732 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:46.733 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:46.733 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:46.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.731+0000 7f5f26ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:46.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.734+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924812651 0 0) 0x7f5f38138d10 con 0x7f5f3813f710 2024-03-25T18:28:46.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.734+0000 7f5f26ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:46.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.734+0000 7f5f26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:46.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.734+0000 7f5f26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:46.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.734+0000 7f5f26ffd640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:28:46.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.734+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f381d7730 con 0x7f5f3813f710 2024-03-25T18:28:46.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.746+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3318203310 0 0) 0x7f5f18004eb0 con 0x7f5f381430c0 2024-03-25T18:28:46.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.746+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f38138d10 con 0x7f5f381430c0 2024-03-25T18:28:46.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.746+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5f30001c30 con 0x7f5f381430c0 2024-03-25T18:28:46.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.746+0000 7f5f26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:46.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.746+0000 7f5f26ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:46.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.746+0000 7f5f26ffd640 10 monclient(hunting): dump: 2024-03-25T18:28:46.747 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:46.747 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:46.747 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:46.747 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:46.747 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:46.747 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:46.747 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:46.748 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:46.748 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:46.748 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:46.751 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1795389926 0 0) 0x7f5f38139900 con 0x7f5f380a98f0 2024-03-25T18:28:46.751 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 >> v1:172.21.15.190:6789/0 conn(0x7f5f381430c0 legacy=0x7f5f381d47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:46.751 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 >> v1:172.21.15.99:6789/0 conn(0x7f5f3813f710 legacy=0x7f5f381d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:46.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f26ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:46.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f26ffd640 1 monclient: found mon.c 2024-03-25T18:28:46.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:46.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f26ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:28:46.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f381d8320 con 0x7f5f380a98f0 2024-03-25T18:28:46.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f26ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:46.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f26ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:46.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:46.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f3caf1640 5 monclient: authenticate success, global_id 4370 2024-03-25T18:28:46.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f3caf1640 10 monclient: _renew_subs 2024-03-25T18:28:46.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f3caf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:28:46.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f3caf1640 1 -- 172.21.15.99:0/1851844553 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f381d41e0 con 0x7f5f380a98f0 2024-03-25T18:28:46.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f3caf1640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:46.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f3caf1640 10 monclient: _renew_subs 2024-03-25T18:28:46.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f3caf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:28:46.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f3caf1640 1 -- 172.21.15.99:0/1851844553 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f381d50e0 con 0x7f5f380a98f0 2024-03-25T18:28:46.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.750+0000 7f5f3caf1640 1 librados: init done 2024-03-25T18:28:46.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.751+0000 7f5f3caf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:46.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.751+0000 7f5f3caf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:46.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.751+0000 7f5f3caf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:28:46.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:46.751+0000 7f5f3caf1640 1 -- 172.21.15.99:0/1851844553 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f380627b0 con 0x7f5f380a98f0 2024-03-25T18:28:47.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.086+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f28001d50 con 0x7f5f380a98f0 2024-03-25T18:28:47.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.086+0000 7f5f26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:47.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.086+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5f28003790 con 0x7f5f380a98f0 2024-03-25T18:28:47.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.086+0000 7f5f26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:47.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.086+0000 7f5f26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:47.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.086+0000 7f5f26ffd640 10 monclient: dump: 2024-03-25T18:28:47.088 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:47.088 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:47.088 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:47.089 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:47.089 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:47.089 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:47.089 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:47.089 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:47.089 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:47.089 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:47.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.086+0000 7f5f34d99640 4 set_mon_vals no callback set 2024-03-25T18:28:47.090 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.087+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f5f280039b0 con 0x7f5f380a98f0 2024-03-25T18:28:47.090 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.088+0000 7f5f35d9b640 10 monclient: get_auth_request con 0x7f5f1806dfb0 auth_method 0 2024-03-25T18:28:47.090 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.088+0000 7f5f26ffd640 10 client.4370.objecter ms_handle_connect 0x7f5f1806dfb0 2024-03-25T18:28:47.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.093+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f5f28053570 con 0x7f5f380a98f0 2024-03-25T18:28:47.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.093+0000 7f5f26ffd640 10 client.4370.objecter ms_dispatch 0x7f5f38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:47.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.093+0000 7f5f26ffd640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:47.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.093+0000 7f5f26ffd640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:47.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.093+0000 7f5f26ffd640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-25T18:28:47.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.093+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f2804eeb0 con 0x7f5f380a98f0 2024-03-25T18:28:47.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.093+0000 7f5f26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:47.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.093+0000 7f5f26ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:47.315 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.314+0000 7f5f3caf1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:47.315 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.314+0000 7f5f3caf1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:47.315 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.314+0000 7f5f3caf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:28:47.316 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.314+0000 7f5f3caf1640 1 -- 172.21.15.99:0/1851844553 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f38136580 con 0x7f5f380a98f0 2024-03-25T18:28:47.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.338+0000 7f5f26ffd640 1 -- 172.21.15.99:0/1851844553 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f5f28050fb0 con 0x7f5f380a98f0 2024-03-25T18:28:47.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.338+0000 7f5f26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:47.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.338+0000 7f5f26ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:47.339 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:28:47.339 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:47.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.341+0000 7f5f3caf1640 10 librados: watch_flush enter 2024-03-25T18:28:47.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.341+0000 7f5f3caf1640 10 librados: watch_flush exit 2024-03-25T18:28:47.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.341+0000 7f5f3caf1640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-25T18:28:47.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.341+0000 7f5f3caf1640 10 client.4370.objecter successfully canceled tick 2024-03-25T18:28:47.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.341+0000 7f5f3caf1640 1 -- 172.21.15.99:0/1851844553 >> v1:172.21.15.190:6800/4220409699 conn(0x7f5f1806dfb0 legacy=0x7f5f18070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:47.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.341+0000 7f5f3caf1640 10 monclient: shutdown 2024-03-25T18:28:47.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.341+0000 7f5f3caf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:47.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.341+0000 7f5f3caf1640 1 -- 172.21.15.99:0/1851844553 >> v1:172.21.15.99:6790/0 conn(0x7f5f380a98f0 legacy=0x7f5f38138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:47.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.341+0000 7f5f3caf1640 1 -- 172.21.15.99:0/1851844553 shutdown_connections 2024-03-25T18:28:47.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.341+0000 7f5f3caf1640 1 -- 172.21.15.99:0/1851844553 >> 172.21.15.99:0/1851844553 conn(0x7f5f380b7ea0 msgr2=0x7f5f38074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:47.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.341+0000 7f5f3caf1640 1 -- 172.21.15.99:0/1851844553 shutdown_connections 2024-03-25T18:28:47.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.341+0000 7f5f3caf1640 1 -- 172.21.15.99:0/1851844553 wait complete. 2024-03-25T18:28:47.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:47.342+0000 7f5f3caf1640 1 librados: shutdown 2024-03-25T18:28:47.354 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:48.945 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:03:49.538 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:48.949 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:03:49.542 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:48.971 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:03:49.564 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:49.019 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:03:49.611 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:49.023 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:03:49.616 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:49.031 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:03:49.589 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:49.040 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:03:49.599 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:49.061 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:03:49.654 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:49.065 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:03:49.624 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:54.160 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:54.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.263+0000 7fd2b2855640 10 monclient: get_monmap_and_config 2024-03-25T18:28:54.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.263+0000 7fd2b2855640 10 monclient: build_initial_monmap 2024-03-25T18:28:54.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.264+0000 7fd2b2855640 10 monclient: monmap: 2024-03-25T18:28:54.265 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:54.265 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:54.265 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:28:54.265023+0000 2024-03-25T18:28:54.265 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:28:54.265023+0000 2024-03-25T18:28:54.265 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:54.265 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:54.265 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:54.265 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:28:54.265 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:28:54.265 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:54.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.264+0000 7fd2b2855640 1 Processor -- start 2024-03-25T18:28:54.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.264+0000 7fd2b2855640 1 -- start start 2024-03-25T18:28:54.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.264+0000 7fd2b2855640 10 monclient: init 2024-03-25T18:28:54.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.265+0000 7fd2b2855640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:54.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.265+0000 7fd2b2855640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:54.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.265+0000 7fd2b2855640 10 monclient(hunting): picked mon.noname-c con 0x7fd2ac0aead0 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:54.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.265+0000 7fd2b2855640 10 monclient(hunting): picked mon.noname-a con 0x7fd2ac135c10 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:54.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.265+0000 7fd2b2855640 10 monclient(hunting): picked mon.noname-b con 0x7fd2ac06f3d0 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:54.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.265+0000 7fd2b2855640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ac071be0 con 0x7fd2ac135c10 2024-03-25T18:28:54.268 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.265+0000 7fd2b2855640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ac06fb30 con 0x7fd2ac0aead0 2024-03-25T18:28:54.268 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.265+0000 7fd2b2855640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ac06f840 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.268 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.265+0000 7fd2b2855640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:54.268 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.265+0000 7fd2b2855640 10 monclient(hunting): authenticate will time out at 7552.433672s 2024-03-25T18:28:54.268 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.266+0000 7fd2abfff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fd2ac0aead0 0x7fd2ac135500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:50094/0 (socket says 172.21.15.99:50094) 2024-03-25T18:28:54.268 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.266+0000 7fd2abfff640 1 -- 172.21.15.99:0/944402858 learned_addr learned my addr 172.21.15.99:0/944402858 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:54.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.336+0000 7fd2aaffd640 1 -- 172.21.15.99:0/944402858 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137611475 0 0) 0x7fd2ac071be0 con 0x7fd2ac135c10 2024-03-25T18:28:54.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.336+0000 7fd2aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:54.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.336+0000 7fd2aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:54.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.336+0000 7fd2aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:54.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.336+0000 7fd2aaffd640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:28:54.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.336+0000 7fd2aaffd640 1 -- 172.21.15.99:0/944402858 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd294003520 con 0x7fd2ac135c10 2024-03-25T18:28:54.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.344+0000 7fd2aaffd640 1 -- 172.21.15.99:0/944402858 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166727971 0 0) 0x7fd2ac06fb30 con 0x7fd2ac0aead0 2024-03-25T18:28:54.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.344+0000 7fd2aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:54.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.344+0000 7fd2aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:54.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.344+0000 7fd2aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:54.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.344+0000 7fd2aaffd640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:28:54.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.344+0000 7fd2aaffd640 1 -- 172.21.15.99:0/944402858 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ac071be0 con 0x7fd2ac0aead0 2024-03-25T18:28:54.356 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 1 -- 172.21.15.99:0/944402858 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2867211565 0 0) 0x7fd2ac071be0 con 0x7fd2ac0aead0 2024-03-25T18:28:54.356 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 1 -- 172.21.15.99:0/944402858 >> v1:172.21.15.99:6790/0 conn(0x7fd2ac06f3d0 legacy=0x7fd2ac072710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:54.356 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 1 -- 172.21.15.99:0/944402858 >> v1:172.21.15.99:6789/0 conn(0x7fd2ac135c10 legacy=0x7fd2ac072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:54.356 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:54.356 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 1 monclient: found mon.noname-c 2024-03-25T18:28:54.356 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:54.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:28:54.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 1 -- 172.21.15.99:0/944402858 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac0b0200 con 0x7fd2ac0aead0 2024-03-25T18:28:54.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:54.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:54.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:54.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 1 -- 172.21.15.99:0/944402858 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fd29c001670 con 0x7fd2ac0aead0 2024-03-25T18:28:54.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:54.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:54.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2aaffd640 10 monclient: dump: 2024-03-25T18:28:54.357 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:54.357 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:54.357 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:54.357 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:54.357 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:54.358 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:54.358 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:54.358 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:54.358 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:54.358 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:54.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2b2855640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:28:54.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.355+0000 7fd2b2855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:54.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.368+0000 7fd2aaffd640 1 -- 172.21.15.99:0/944402858 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd29c0017d0 con 0x7fd2ac0aead0 2024-03-25T18:28:54.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.368+0000 7fd2aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:54.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.368+0000 7fd2aaffd640 1 -- 172.21.15.99:0/944402858 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fd29c001ba0 con 0x7fd2ac0aead0 2024-03-25T18:28:54.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.368+0000 7fd2aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:54.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.368+0000 7fd2aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:54.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.368+0000 7fd2aaffd640 10 monclient: dump: 2024-03-25T18:28:54.372 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:54.372 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:54.372 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:54.372 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:54.372 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:54.372 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:54.372 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:54.372 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:54.372 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:54.372 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:54.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.368+0000 7fd2b2855640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:54.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.368+0000 7fd2b2855640 4 set_mon_vals no callback set 2024-03-25T18:28:54.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.369+0000 7fd2b2855640 10 monclient: shutdown 2024-03-25T18:28:54.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.369+0000 7fd2b2855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.369+0000 7fd2b2855640 1 -- 172.21.15.99:0/944402858 >> v1:172.21.15.190:6789/0 conn(0x7fd2ac0aead0 legacy=0x7fd2ac135500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.369+0000 7fd2b2855640 1 -- 172.21.15.99:0/944402858 shutdown_connections 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.369+0000 7fd2b2855640 1 -- 172.21.15.99:0/944402858 >> 172.21.15.99:0/944402858 conn(0x7fd2ac0b3e60 msgr2=0x7fd2ac0b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.369+0000 7fd2b2855640 1 -- 172.21.15.99:0/944402858 shutdown_connections 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.369+0000 7fd2b2855640 1 -- 172.21.15.99:0/944402858 wait complete. 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.369+0000 7fd2b2855640 10 monclient: build_initial_monmap 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.369+0000 7fd2b2855640 10 monclient: monmap: 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:28:54.373 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:28:54.374 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:28:54.374 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:54.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.370+0000 7fd2b2855640 1 librados: starting msgr at 2024-03-25T18:28:54.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.370+0000 7fd2b2855640 1 librados: starting objecter 2024-03-25T18:28:54.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.370+0000 7fd2b2855640 1 Processor -- start 2024-03-25T18:28:54.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.370+0000 7fd2b2855640 1 -- start start 2024-03-25T18:28:54.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.370+0000 7fd2b2855640 1 librados: setting wanted keys 2024-03-25T18:28:54.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.370+0000 7fd2b2855640 1 librados: calling monclient init 2024-03-25T18:28:54.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.370+0000 7fd2b2855640 10 monclient: init 2024-03-25T18:28:54.375 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.371+0000 7fd2b2855640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:54.375 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.371+0000 7fd2b2855640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:54.375 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.371+0000 7fd2b2855640 10 monclient(hunting): picked mon.noname-b con 0x7fd2ac06f3d0 addr v1:172.21.15.190:6789/0 2024-03-25T18:28:54.376 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.371+0000 7fd2b2855640 10 monclient(hunting): picked mon.noname-a con 0x7fd2ac0aead0 addr v1:172.21.15.99:6789/0 2024-03-25T18:28:54.376 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.371+0000 7fd2b2855640 10 monclient(hunting): picked mon.noname-c con 0x7fd2ac135c10 addr v1:172.21.15.99:6790/0 2024-03-25T18:28:54.376 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.371+0000 7fd2b2855640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ac13ea80 con 0x7fd2ac0aead0 2024-03-25T18:28:54.376 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.371+0000 7fd2b2855640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ac138c30 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.376 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.371+0000 7fd2b2855640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ac13d2d0 con 0x7fd2ac135c10 2024-03-25T18:28:54.376 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.371+0000 7fd2b2855640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:54.376 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.371+0000 7fd2b2855640 10 monclient(hunting): authenticate will time out at 7552.539160s 2024-03-25T18:28:54.382 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.381+0000 7fd2abfff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fd2ac06f3d0 0x7fd2ac1dab50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:50118/0 (socket says 172.21.15.99:50118) 2024-03-25T18:28:54.382 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.381+0000 7fd2abfff640 1 -- 172.21.15.99:0/574169018 learned_addr learned my addr 172.21.15.99:0/574169018 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:28:54.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.382+0000 7fd2a8ff9640 10 client.?.objecter ms_handle_connect 0x7fd2ac06f3d0 2024-03-25T18:28:54.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.382+0000 7fd2a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:54.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.388+0000 7fd2a8ff9640 10 client.?.objecter ms_handle_connect 0x7fd2ac135c10 2024-03-25T18:28:54.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.388+0000 7fd2a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:54.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.389+0000 7fd2a8ff9640 10 client.?.objecter ms_handle_connect 0x7fd2ac0aead0 2024-03-25T18:28:54.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.389+0000 7fd2a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:54.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.400+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912203513 0 0) 0x7fd2ac138c30 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.400+0000 7fd2a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:54.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.400+0000 7fd2a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:54.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.400+0000 7fd2a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:54.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.400+0000 7fd2a8ff9640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:28:54.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.400+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd298004eb0 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.402+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173188370 0 0) 0x7fd2ac13d2d0 con 0x7fd2ac135c10 2024-03-25T18:28:54.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.402+0000 7fd2a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:54.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.402+0000 7fd2a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:54.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.402+0000 7fd2a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:54.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.402+0000 7fd2a8ff9640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:28:54.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.403+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ac138c30 con 0x7fd2ac135c10 2024-03-25T18:28:54.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.408+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689634353 0 0) 0x7fd2ac13ea80 con 0x7fd2ac0aead0 2024-03-25T18:28:54.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.408+0000 7fd2a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:54.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.408+0000 7fd2a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:54.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.408+0000 7fd2a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:54.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.408+0000 7fd2a8ff9640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:28:54.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.408+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ac13d2d0 con 0x7fd2ac0aead0 2024-03-25T18:28:54.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.425+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3320701287 0 0) 0x7fd298004eb0 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.425+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2ac13ea80 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.425+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fd29c001be0 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.425+0000 7fd2a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:54.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.425+0000 7fd2a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:54.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.425+0000 7fd2a8ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:54.426 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:54.427 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:54.427 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:54.427 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:54.427 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:54.427 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:54.427 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:54.427 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:54.427 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:54.427 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:54.427 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.425+0000 7fd2a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:54.428 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.426+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2340303293 0 0) 0x7fd2ac13d2d0 con 0x7fd2ac0aead0 2024-03-25T18:28:54.428 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.426+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd298004eb0 con 0x7fd2ac0aead0 2024-03-25T18:28:54.428 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.426+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fd2a0001850 con 0x7fd2ac0aead0 2024-03-25T18:28:54.428 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.426+0000 7fd2a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:54.428 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.426+0000 7fd2a8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:54.428 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.426+0000 7fd2a8ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:54.428 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:54.428 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:54.428 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:54.428 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:54.428 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:54.429 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:54.429 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:54.429 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:54.429 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:54.429 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:54.437 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.435+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3168660542 0 0) 0x7fd2ac138c30 con 0x7fd2ac135c10 2024-03-25T18:28:54.437 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.435+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2ac13d2d0 con 0x7fd2ac135c10 2024-03-25T18:28:54.437 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.435+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fd2a4001ba0 con 0x7fd2ac135c10 2024-03-25T18:28:54.437 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.435+0000 7fd2a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:54.437 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.436+0000 7fd2a8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:54.437 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.436+0000 7fd2a8ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:54.437 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:54.437 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:54.437 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:54.437 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:54.437 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:54.437 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:54.438 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:54.438 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:54.438 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:54.438 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:54.446 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1738437538 0 0) 0x7fd2ac13ea80 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.446 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 >> v1:172.21.15.99:6790/0 conn(0x7fd2ac135c10 legacy=0x7fd2ac12b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:54.446 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 >> v1:172.21.15.99:6789/0 conn(0x7fd2ac0aead0 legacy=0x7fd2ac12ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:54.446 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2a8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:54.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2a8ff9640 1 monclient: found mon.b 2024-03-25T18:28:54.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:54.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2a8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:54.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac13d7f0 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2a8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:54.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2a8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:54.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:54.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2b2855640 5 monclient: authenticate success, global_id 4384 2024-03-25T18:28:54.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2b2855640 10 monclient: _renew_subs 2024-03-25T18:28:54.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2b2855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:54.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.444+0000 7fd2b2855640 1 -- 172.21.15.99:0/574169018 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ac138e90 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.445+0000 7fd2b2855640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:54.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.445+0000 7fd2b2855640 10 monclient: _renew_subs 2024-03-25T18:28:54.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.445+0000 7fd2b2855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:54.448 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.445+0000 7fd2b2855640 1 -- 172.21.15.99:0/574169018 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ac13d020 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.448 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.445+0000 7fd2b2855640 1 librados: init done 2024-03-25T18:28:54.448 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.446+0000 7fd2b2855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:54.448 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.446+0000 7fd2b2855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:54.448 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.446+0000 7fd2b2855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:54.448 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.446+0000 7fd2b2855640 1 -- 172.21.15.99:0/574169018 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ac13d460 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.722+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd29c004a90 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.722+0000 7fd2a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:54.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.722+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fd29c004e60 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.722+0000 7fd2a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:54.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.722+0000 7fd2a8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:54.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.722+0000 7fd2a8ff9640 10 monclient: dump: 2024-03-25T18:28:54.725 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:28:54.725 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:28:54.725 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:54.725 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:28:54.725 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:54.725 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:28:54.725 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:28:54.725 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:28:54.725 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:28:54.725 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:28:54.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.722+0000 7fd2aa7fc640 4 set_mon_vals no callback set 2024-03-25T18:28:54.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.723+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fd29c005080 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.724+0000 7fd2ab7fe640 10 monclient: get_auth_request con 0x7fd29806e0f0 auth_method 0 2024-03-25T18:28:54.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.724+0000 7fd2a8ff9640 10 client.4384.objecter ms_handle_connect 0x7fd29806e0f0 2024-03-25T18:28:54.730 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.728+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fd29c054990 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.730 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.729+0000 7fd2a8ff9640 10 client.4384.objecter ms_dispatch 0x7fd2ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:54.730 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.729+0000 7fd2a8ff9640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:54.730 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.729+0000 7fd2a8ff9640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:54.730 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.729+0000 7fd2a8ff9640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-25T18:28:54.730 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.729+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd29c0504a0 con 0x7fd2ac06f3d0 2024-03-25T18:28:54.730 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.729+0000 7fd2a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:54.731 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.729+0000 7fd2a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:54.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.974+0000 7fd2b2855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:54.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.975+0000 7fd2b2855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:54.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.975+0000 7fd2b2855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:28:54.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:54.975+0000 7fd2b2855640 1 -- 172.21.15.99:0/574169018 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2ac0627b0 con 0x7fd2ac06f3d0 2024-03-25T18:28:55.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.016+0000 7fd2a8ff9640 1 -- 172.21.15.99:0/574169018 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fd29c0523d0 con 0x7fd2ac06f3d0 2024-03-25T18:28:55.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.016+0000 7fd2a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:55.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.016+0000 7fd2a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:55.017 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:28:55.017 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:55.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.018+0000 7fd2b2855640 10 librados: watch_flush enter 2024-03-25T18:28:55.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.018+0000 7fd2b2855640 10 librados: watch_flush exit 2024-03-25T18:28:55.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.018+0000 7fd2b2855640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-25T18:28:55.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.018+0000 7fd2b2855640 10 client.4384.objecter successfully canceled tick 2024-03-25T18:28:55.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.019+0000 7fd2b2855640 1 -- 172.21.15.99:0/574169018 >> v1:172.21.15.190:6800/4220409699 conn(0x7fd29806e0f0 legacy=0x7fd2980705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:55.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.019+0000 7fd2b2855640 10 monclient: shutdown 2024-03-25T18:28:55.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.019+0000 7fd2b2855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:55.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.019+0000 7fd2b2855640 1 -- 172.21.15.99:0/574169018 >> v1:172.21.15.190:6789/0 conn(0x7fd2ac06f3d0 legacy=0x7fd2ac1dab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:55.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.019+0000 7fd2b2855640 1 -- 172.21.15.99:0/574169018 shutdown_connections 2024-03-25T18:28:55.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.019+0000 7fd2b2855640 1 -- 172.21.15.99:0/574169018 >> 172.21.15.99:0/574169018 conn(0x7fd2ac0b3e60 msgr2=0x7fd2ac0b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:55.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.019+0000 7fd2b2855640 1 -- 172.21.15.99:0/574169018 shutdown_connections 2024-03-25T18:28:55.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.019+0000 7fd2b2855640 1 -- 172.21.15.99:0/574169018 wait complete. 2024-03-25T18:28:55.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:28:55.019+0000 7fd2b2855640 1 librados: shutdown 2024-03-25T18:28:55.033 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:01.842 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:01.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.954+0000 7f5f3c6ae640 10 monclient: get_monmap_and_config 2024-03-25T18:29:01.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.954+0000 7f5f3c6ae640 10 monclient: build_initial_monmap 2024-03-25T18:29:01.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.954+0000 7f5f3c6ae640 10 monclient: monmap: 2024-03-25T18:29:01.956 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:01.956 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:01.956 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:29:01.955667+0000 2024-03-25T18:29:01.956 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:29:01.955667+0000 2024-03-25T18:29:01.956 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:01.956 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:01.956 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:01.956 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:29:01.956 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:29:01.956 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:01.957 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.955+0000 7f5f3c6ae640 1 Processor -- start 2024-03-25T18:29:01.958 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.955+0000 7f5f3c6ae640 1 -- start start 2024-03-25T18:29:01.958 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.955+0000 7f5f3c6ae640 10 monclient: init 2024-03-25T18:29:01.958 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.956+0000 7f5f3c6ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:01.958 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.956+0000 7f5f3c6ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:01.958 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.956+0000 7f5f3c6ae640 10 monclient(hunting): picked mon.noname-a con 0x7f5f340b4d60 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:01.958 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.956+0000 7f5f3c6ae640 10 monclient(hunting): picked mon.noname-c con 0x7f5f340b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:01.958 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.956+0000 7f5f3c6ae640 10 monclient(hunting): picked mon.noname-b con 0x7f5f3414be90 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:01.958 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.956+0000 7f5f3c6ae640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f340af940 con 0x7f5f340b4d60 2024-03-25T18:29:01.958 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.956+0000 7f5f3c6ae640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f340b0a70 con 0x7f5f340b1730 2024-03-25T18:29:01.958 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.956+0000 7f5f3c6ae640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f340b0d50 con 0x7f5f3414be90 2024-03-25T18:29:01.958 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.956+0000 7f5f3c6ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:01.958 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.956+0000 7f5f3c6ae640 10 monclient(hunting): authenticate will time out at 7560.123988s 2024-03-25T18:29:01.958 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.956+0000 7f5f3ac4b640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f5f3414be90 0x7f5f3414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:54158/0 (socket says 172.21.15.99:54158) 2024-03-25T18:29:01.959 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:01.956+0000 7f5f3ac4b640 1 -- 172.21.15.99:0/229105686 learned_addr learned my addr 172.21.15.99:0/229105686 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:02.038 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.037+0000 7f5f39448640 1 -- 172.21.15.99:0/229105686 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100108098 0 0) 0x7f5f340af940 con 0x7f5f340b4d60 2024-03-25T18:29:02.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.037+0000 7f5f39448640 10 monclient(hunting): _negotiate 2024-03-25T18:29:02.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.037+0000 7f5f39448640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:02.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.037+0000 7f5f39448640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:02.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.037+0000 7f5f39448640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:29:02.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.037+0000 7f5f39448640 1 -- 172.21.15.99:0/229105686 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f20003520 con 0x7f5f340b4d60 2024-03-25T18:29:02.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 1 -- 172.21.15.99:0/229105686 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1149255593 0 0) 0x7f5f20003520 con 0x7f5f340b4d60 2024-03-25T18:29:02.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 1 -- 172.21.15.99:0/229105686 >> v1:172.21.15.99:6790/0 conn(0x7f5f3414be90 legacy=0x7f5f3414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:02.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 1 -- 172.21.15.99:0/229105686 >> v1:172.21.15.190:6789/0 conn(0x7f5f340b1730 legacy=0x7f5f340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:02.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 10 monclient: _finish_hunting 0 2024-03-25T18:29:02.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 1 monclient: found mon.noname-a 2024-03-25T18:29:02.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:02.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 1 -- 172.21.15.99:0/229105686 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f340b1030 con 0x7f5f340b4d60 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 10 monclient: _finish_auth 0 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 10 monclient: _check_auth_tickets 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 1 -- 172.21.15.99:0/229105686 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5f30001690 con 0x7f5f340b4d60 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f39448640 10 monclient: dump: 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:02.061 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:02.062 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:02.062 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:02.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f3c6ae640 5 monclient: authenticate success, global_id 4371 2024-03-25T18:29:02.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.059+0000 7f5f3c6ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:02.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.078+0000 7f5f39448640 1 -- 172.21.15.99:0/229105686 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f300017f0 con 0x7f5f340b4d60 2024-03-25T18:29:02.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.078+0000 7f5f39448640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:02.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.078+0000 7f5f39448640 1 -- 172.21.15.99:0/229105686 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5f30001c00 con 0x7f5f340b4d60 2024-03-25T18:29:02.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.078+0000 7f5f3c6ae640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:02.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.078+0000 7f5f3c6ae640 4 set_mon_vals no callback set 2024-03-25T18:29:02.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.078+0000 7f5f39448640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:02.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.078+0000 7f5f39448640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:02.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.078+0000 7f5f39448640 10 monclient: dump: 2024-03-25T18:29:02.080 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:02.080 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:02.080 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:02.080 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:02.080 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:02.080 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:02.080 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:02.080 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:02.080 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:02.080 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:02.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.079+0000 7f5f3c6ae640 10 monclient: shutdown 2024-03-25T18:29:02.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.079+0000 7f5f3c6ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:02.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.079+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/229105686 >> v1:172.21.15.99:6789/0 conn(0x7f5f340b4d60 legacy=0x7f5f3413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:02.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.079+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/229105686 shutdown_connections 2024-03-25T18:29:02.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.079+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/229105686 >> 172.21.15.99:0/229105686 conn(0x7f5f340617a0 msgr2=0x7f5f34061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.079+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/229105686 shutdown_connections 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.079+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/229105686 wait complete. 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.079+0000 7f5f3c6ae640 10 monclient: build_initial_monmap 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.080+0000 7f5f3c6ae640 10 monclient: monmap: 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:02.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.080+0000 7f5f3c6ae640 1 librados: starting msgr at 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.080+0000 7f5f3c6ae640 1 librados: starting objecter 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.080+0000 7f5f3c6ae640 1 Processor -- start 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.080+0000 7f5f3c6ae640 1 -- start start 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.080+0000 7f5f3c6ae640 1 librados: setting wanted keys 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.080+0000 7f5f3c6ae640 1 librados: calling monclient init 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.080+0000 7f5f3c6ae640 10 monclient: init 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.081+0000 7f5f3c6ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.081+0000 7f5f3c6ae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.081+0000 7f5f3c6ae640 10 monclient(hunting): picked mon.noname-a con 0x7f5f340b1730 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.081+0000 7f5f3c6ae640 10 monclient(hunting): picked mon.noname-c con 0x7f5f340b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.081+0000 7f5f3c6ae640 10 monclient(hunting): picked mon.noname-b con 0x7f5f3414be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.081+0000 7f5f3c6ae640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f341d8200 con 0x7f5f340b1730 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.081+0000 7f5f3c6ae640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f34139250 con 0x7f5f3414be90 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.081+0000 7f5f3c6ae640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f341362d0 con 0x7f5f340b4d60 2024-03-25T18:29:02.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.081+0000 7f5f3c6ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:02.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.081+0000 7f5f3c6ae640 10 monclient(hunting): authenticate will time out at 7560.249340s 2024-03-25T18:29:02.090 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.089+0000 7f5f3ac4b640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f5f3414be90 0x7f5f341e2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:54460/0 (socket says 172.21.15.99:54460) 2024-03-25T18:29:02.090 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.089+0000 7f5f3ac4b640 1 -- 172.21.15.99:0/730370835 learned_addr learned my addr 172.21.15.99:0/730370835 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:02.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.092+0000 7f5f2affd640 10 client.?.objecter ms_handle_connect 0x7f5f3414be90 2024-03-25T18:29:02.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.092+0000 7f5f2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:02.097 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.096+0000 7f5f2affd640 10 client.?.objecter ms_handle_connect 0x7f5f340b4d60 2024-03-25T18:29:02.097 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.096+0000 7f5f2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:02.106 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.105+0000 7f5f2affd640 10 client.?.objecter ms_handle_connect 0x7f5f340b1730 2024-03-25T18:29:02.106 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.105+0000 7f5f2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:02.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.107+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235964700 0 0) 0x7f5f341362d0 con 0x7f5f340b4d60 2024-03-25T18:29:02.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.107+0000 7f5f2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:02.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.107+0000 7f5f2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:02.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.107+0000 7f5f2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:02.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.107+0000 7f5f2affd640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:29:02.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.107+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f18004eb0 con 0x7f5f340b4d60 2024-03-25T18:29:02.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.107+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293293687 0 0) 0x7f5f34139250 con 0x7f5f3414be90 2024-03-25T18:29:02.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.107+0000 7f5f2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:02.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.107+0000 7f5f2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:02.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.107+0000 7f5f2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:02.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.107+0000 7f5f2affd640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:29:02.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.107+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f341362d0 con 0x7f5f3414be90 2024-03-25T18:29:02.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.115+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207323821 0 0) 0x7f5f341d8200 con 0x7f5f340b1730 2024-03-25T18:29:02.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.115+0000 7f5f2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:02.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.115+0000 7f5f2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:02.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.115+0000 7f5f2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:02.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.115+0000 7f5f2affd640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:29:02.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.115+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f34139250 con 0x7f5f340b1730 2024-03-25T18:29:02.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.120+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 257101072 0 0) 0x7f5f18004eb0 con 0x7f5f340b4d60 2024-03-25T18:29:02.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.120+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f341d8200 con 0x7f5f340b4d60 2024-03-25T18:29:02.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.120+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5f240018c0 con 0x7f5f340b4d60 2024-03-25T18:29:02.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.120+0000 7f5f2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:02.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.120+0000 7f5f2affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:02.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.120+0000 7f5f2affd640 10 monclient(hunting): dump: 2024-03-25T18:29:02.123 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:02.123 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:02.123 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:02.123 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:02.123 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:02.124 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:02.124 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:02.124 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:02.124 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:02.124 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:02.124 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.120+0000 7f5f2affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:02.133 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.131+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3086644272 0 0) 0x7f5f341362d0 con 0x7f5f3414be90 2024-03-25T18:29:02.133 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.131+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f18004eb0 con 0x7f5f3414be90 2024-03-25T18:29:02.133 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.131+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5f2c001b50 con 0x7f5f3414be90 2024-03-25T18:29:02.133 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.131+0000 7f5f2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:02.133 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.131+0000 7f5f2affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:02.133 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.131+0000 7f5f2affd640 10 monclient(hunting): dump: 2024-03-25T18:29:02.134 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:02.134 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:02.134 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:02.134 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:02.134 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:02.134 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:02.134 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:02.134 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:02.135 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:02.135 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:02.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.131+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4183362232 0 0) 0x7f5f34139250 con 0x7f5f340b1730 2024-03-25T18:29:02.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.131+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f341362d0 con 0x7f5f340b1730 2024-03-25T18:29:02.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.131+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5f30001bc0 con 0x7f5f340b1730 2024-03-25T18:29:02.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.131+0000 7f5f2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:02.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.131+0000 7f5f2affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:02.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.131+0000 7f5f2affd640 10 monclient(hunting): dump: 2024-03-25T18:29:02.136 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:02.136 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:02.136 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:02.136 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:02.136 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:02.136 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:02.136 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:02.136 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:02.137 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:02.137 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:02.145 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.143+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3847516189 0 0) 0x7f5f18004eb0 con 0x7f5f3414be90 2024-03-25T18:29:02.145 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.143+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 >> v1:172.21.15.99:6790/0 conn(0x7f5f340b4d60 legacy=0x7f5f341d8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:02.145 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.143+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 >> v1:172.21.15.99:6789/0 conn(0x7f5f340b1730 legacy=0x7f5f34137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:02.145 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.143+0000 7f5f2affd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:02.145 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.143+0000 7f5f2affd640 1 monclient: found mon.b 2024-03-25T18:29:02.145 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.143+0000 7f5f2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:02.145 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.143+0000 7f5f2affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:02.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.143+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3413be10 con 0x7f5f3414be90 2024-03-25T18:29:02.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.143+0000 7f5f2affd640 10 monclient: _finish_auth 0 2024-03-25T18:29:02.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.143+0000 7f5f2affd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:02.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.143+0000 7f5f2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:02.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.143+0000 7f5f3c6ae640 5 monclient: authenticate success, global_id 4390 2024-03-25T18:29:02.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.144+0000 7f5f3c6ae640 10 monclient: _renew_subs 2024-03-25T18:29:02.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.144+0000 7f5f3c6ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:02.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.144+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/730370835 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f34137780 con 0x7f5f3414be90 2024-03-25T18:29:02.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.144+0000 7f5f3c6ae640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:02.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.144+0000 7f5f3c6ae640 10 monclient: _renew_subs 2024-03-25T18:29:02.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.144+0000 7f5f3c6ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:02.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.144+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/730370835 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f34143ae0 con 0x7f5f3414be90 2024-03-25T18:29:02.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.144+0000 7f5f3c6ae640 1 librados: init done 2024-03-25T18:29:02.147 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.145+0000 7f5f3c6ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:02.147 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.145+0000 7f5f3c6ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:02.147 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.145+0000 7f5f3c6ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:02.147 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.145+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/730370835 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f341d83b0 con 0x7f5f3414be90 2024-03-25T18:29:02.356 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.355+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f2c001f80 con 0x7f5f3414be90 2024-03-25T18:29:02.356 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.355+0000 7f5f2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:02.356 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.355+0000 7f5f38c47640 4 set_mon_vals no callback set 2024-03-25T18:29:02.421 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.420+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5f2c004d00 con 0x7f5f3414be90 2024-03-25T18:29:02.421 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.420+0000 7f5f2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:02.421 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.420+0000 7f5f2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:02.421 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.420+0000 7f5f2affd640 10 monclient: dump: 2024-03-25T18:29:02.421 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:02.421 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:02.422 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:02.422 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:02.422 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:02.422 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:02.422 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:02.422 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:02.422 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:02.422 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:02.423 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.421+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f5f2c01acd0 con 0x7f5f3414be90 2024-03-25T18:29:02.423 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.421+0000 7f5f3a44a640 10 monclient: get_auth_request con 0x7f5f1806e110 auth_method 0 2024-03-25T18:29:02.423 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.421+0000 7f5f2affd640 10 client.4390.objecter ms_handle_connect 0x7f5f1806e110 2024-03-25T18:29:02.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.423+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f5f2c054980 con 0x7f5f3414be90 2024-03-25T18:29:02.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.423+0000 7f5f2affd640 10 client.4390.objecter ms_dispatch 0x7f5f34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:02.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.423+0000 7f5f2affd640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:02.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.423+0000 7f5f2affd640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:02.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.423+0000 7f5f2affd640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-25T18:29:02.431 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.429+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f2c0502c0 con 0x7f5f3414be90 2024-03-25T18:29:02.431 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.430+0000 7f5f2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:02.431 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.430+0000 7f5f2affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:02.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.678+0000 7f5f3c6ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:02.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.678+0000 7f5f3c6ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:02.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.678+0000 7f5f3c6ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:02.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.678+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/730370835 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f340627b0 con 0x7f5f3414be90 2024-03-25T18:29:02.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.704+0000 7f5f2affd640 1 -- 172.21.15.99:0/730370835 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f5f2c0523c0 con 0x7f5f3414be90 2024-03-25T18:29:02.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.704+0000 7f5f2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:02.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.704+0000 7f5f2affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:02.706 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:29:02.706 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:02.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.707+0000 7f5f3c6ae640 10 librados: watch_flush enter 2024-03-25T18:29:02.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.707+0000 7f5f3c6ae640 10 librados: watch_flush exit 2024-03-25T18:29:02.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.707+0000 7f5f3c6ae640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-25T18:29:02.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.707+0000 7f5f3c6ae640 10 client.4390.objecter successfully canceled tick 2024-03-25T18:29:02.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.707+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/730370835 >> v1:172.21.15.190:6800/4220409699 conn(0x7f5f1806e110 legacy=0x7f5f180705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:02.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.707+0000 7f5f3c6ae640 10 monclient: shutdown 2024-03-25T18:29:02.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.707+0000 7f5f3c6ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:02.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.707+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/730370835 >> v1:172.21.15.190:6789/0 conn(0x7f5f3414be90 legacy=0x7f5f341e2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:02.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.707+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/730370835 shutdown_connections 2024-03-25T18:29:02.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.707+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/730370835 >> 172.21.15.99:0/730370835 conn(0x7f5f340617a0 msgr2=0x7f5f340b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:02.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.707+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/730370835 shutdown_connections 2024-03-25T18:29:02.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.707+0000 7f5f3c6ae640 1 -- 172.21.15.99:0/730370835 wait complete. 2024-03-25T18:29:02.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:02.707+0000 7f5f3c6ae640 1 librados: shutdown 2024-03-25T18:29:02.719 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:09.015 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:04:09.608 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.023 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:04:09.616 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.059 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:04:09.649 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.070 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:04:09.662 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.075 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:04:09.667 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.081 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:04:09.640 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.093 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:04:09.651 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.108 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:04:09.700 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.133 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:04:09.692 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:09.527 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:09.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.644+0000 7f6e533cf640 10 monclient: get_monmap_and_config 2024-03-25T18:29:09.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.644+0000 7f6e533cf640 10 monclient: build_initial_monmap 2024-03-25T18:29:09.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.644+0000 7f6e533cf640 10 monclient: monmap: 2024-03-25T18:29:09.645 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:09.645 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:09.645 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:29:09.645316+0000 2024-03-25T18:29:09.646 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:29:09.645316+0000 2024-03-25T18:29:09.646 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:09.646 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:09.646 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:09.646 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:29:09.646 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:29:09.646 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:09.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.645+0000 7f6e533cf640 1 Processor -- start 2024-03-25T18:29:09.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.645+0000 7f6e533cf640 1 -- start start 2024-03-25T18:29:09.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.645+0000 7f6e533cf640 10 monclient: init 2024-03-25T18:29:09.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.646+0000 7f6e533cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:09.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.646+0000 7f6e533cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:09.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.646+0000 7f6e533cf640 10 monclient(hunting): picked mon.noname-b con 0x7f6e4c0705b0 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:09.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.646+0000 7f6e533cf640 10 monclient(hunting): picked mon.noname-c con 0x7f6e4c1402d0 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:09.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.646+0000 7f6e533cf640 10 monclient(hunting): picked mon.noname-a con 0x7f6e4c149ae0 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:09.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.646+0000 7f6e533cf640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4c0718c0 con 0x7f6e4c149ae0 2024-03-25T18:29:09.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.646+0000 7f6e533cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4c0714c0 con 0x7f6e4c1402d0 2024-03-25T18:29:09.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.646+0000 7f6e533cf640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4c072b50 con 0x7f6e4c0705b0 2024-03-25T18:29:09.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.646+0000 7f6e533cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:09.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.646+0000 7f6e533cf640 10 monclient(hunting): authenticate will time out at 7567.814133s 2024-03-25T18:29:09.658 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.657+0000 7f6e5116b640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f6e4c0705b0 0x7f6e4c13d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:37612/0 (socket says 172.21.15.99:37612) 2024-03-25T18:29:09.658 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.657+0000 7f6e5116b640 1 -- 172.21.15.99:0/3288640853 learned_addr learned my addr 172.21.15.99:0/3288640853 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:09.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.668+0000 7f6e43fff640 1 -- 172.21.15.99:0/3288640853 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777235969 0 0) 0x7f6e4c072b50 con 0x7f6e4c0705b0 2024-03-25T18:29:09.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.668+0000 7f6e43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:09.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.668+0000 7f6e43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:09.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.668+0000 7f6e43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:09.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.668+0000 7f6e43fff640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:29:09.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.668+0000 7f6e43fff640 1 -- 172.21.15.99:0/3288640853 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e38003520 con 0x7f6e4c0705b0 2024-03-25T18:29:09.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.674+0000 7f6e43fff640 1 -- 172.21.15.99:0/3288640853 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212994617 0 0) 0x7f6e4c0718c0 con 0x7f6e4c149ae0 2024-03-25T18:29:09.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.674+0000 7f6e43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:09.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.674+0000 7f6e43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:09.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.674+0000 7f6e43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:09.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.674+0000 7f6e43fff640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:29:09.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.674+0000 7f6e43fff640 1 -- 172.21.15.99:0/3288640853 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e4c072b50 con 0x7f6e4c149ae0 2024-03-25T18:29:09.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.684+0000 7f6e43fff640 1 -- 172.21.15.99:0/3288640853 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010859839 0 0) 0x7f6e4c0714c0 con 0x7f6e4c1402d0 2024-03-25T18:29:09.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.684+0000 7f6e43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:09.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.684+0000 7f6e43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:09.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.684+0000 7f6e43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:09.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.684+0000 7f6e43fff640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:29:09.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.684+0000 7f6e43fff640 1 -- 172.21.15.99:0/3288640853 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e4c0718c0 con 0x7f6e4c1402d0 2024-03-25T18:29:09.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.688+0000 7f6e43fff640 1 -- 172.21.15.99:0/3288640853 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 750430873 0 0) 0x7f6e38003520 con 0x7f6e4c0705b0 2024-03-25T18:29:09.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.688+0000 7f6e43fff640 1 -- 172.21.15.99:0/3288640853 >> v1:172.21.15.190:6789/0 conn(0x7f6e4c1402d0 legacy=0x7f6e4c140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:09.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e43fff640 1 -- 172.21.15.99:0/3288640853 >> v1:172.21.15.99:6789/0 conn(0x7f6e4c149ae0 legacy=0x7f6e4c14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:09.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e43fff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:09.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e43fff640 1 monclient: found mon.noname-b 2024-03-25T18:29:09.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:09.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e43fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:29:09.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e43fff640 1 -- 172.21.15.99:0/3288640853 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c13dd10 con 0x7f6e4c0705b0 2024-03-25T18:29:09.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e43fff640 10 monclient: _finish_auth 0 2024-03-25T18:29:09.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e43fff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:09.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:09.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e43fff640 1 -- 172.21.15.99:0/3288640853 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6e48001690 con 0x7f6e4c0705b0 2024-03-25T18:29:09.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:09.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e43fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:09.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e43fff640 10 monclient: dump: 2024-03-25T18:29:09.691 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:09.691 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:09.691 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:09.692 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:09.692 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:09.692 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:09.692 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:09.692 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:09.692 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:09.692 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:09.692 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e533cf640 5 monclient: authenticate success, global_id 4385 2024-03-25T18:29:09.692 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.689+0000 7f6e533cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:09.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.706+0000 7f6e43fff640 1 -- 172.21.15.99:0/3288640853 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e480017f0 con 0x7f6e4c0705b0 2024-03-25T18:29:09.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.706+0000 7f6e43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:09.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.706+0000 7f6e43fff640 1 -- 172.21.15.99:0/3288640853 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6e48001c00 con 0x7f6e4c0705b0 2024-03-25T18:29:09.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.706+0000 7f6e43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:09.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.706+0000 7f6e43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:09.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.706+0000 7f6e43fff640 10 monclient: dump: 2024-03-25T18:29:09.708 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:09.708 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:09.708 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:09.708 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:09.708 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:09.708 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:09.708 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:09.708 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:09.708 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:09.708 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:09.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.706+0000 7f6e533cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:09.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.706+0000 7f6e533cf640 4 set_mon_vals no callback set 2024-03-25T18:29:09.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.707+0000 7f6e533cf640 10 monclient: shutdown 2024-03-25T18:29:09.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.707+0000 7f6e533cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:09.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.707+0000 7f6e533cf640 1 -- 172.21.15.99:0/3288640853 >> v1:172.21.15.99:6790/0 conn(0x7f6e4c0705b0 legacy=0x7f6e4c13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:09.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.707+0000 7f6e533cf640 1 -- 172.21.15.99:0/3288640853 shutdown_connections 2024-03-25T18:29:09.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.707+0000 7f6e533cf640 1 -- 172.21.15.99:0/3288640853 >> 172.21.15.99:0/3288640853 conn(0x7f6e4c0b5d10 msgr2=0x7f6e4c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:09.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.707+0000 7f6e533cf640 1 -- 172.21.15.99:0/3288640853 shutdown_connections 2024-03-25T18:29:09.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.707+0000 7f6e533cf640 1 -- 172.21.15.99:0/3288640853 wait complete. 2024-03-25T18:29:09.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.708+0000 7f6e533cf640 10 monclient: build_initial_monmap 2024-03-25T18:29:09.711 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.708+0000 7f6e533cf640 10 monclient: monmap: 2024-03-25T18:29:09.711 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:09.711 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:09.711 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:29:09.711 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.708+0000 7f6e533cf640 1 librados: starting msgr at 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.708+0000 7f6e533cf640 1 librados: starting objecter 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.708+0000 7f6e533cf640 1 Processor -- start 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.708+0000 7f6e533cf640 1 -- start start 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.708+0000 7f6e533cf640 1 librados: setting wanted keys 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.708+0000 7f6e533cf640 1 librados: calling monclient init 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.708+0000 7f6e533cf640 10 monclient: init 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.709+0000 7f6e533cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.709+0000 7f6e533cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:09.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.709+0000 7f6e533cf640 10 monclient(hunting): picked mon.noname-c con 0x7f6e4c0705b0 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:09.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.709+0000 7f6e533cf640 10 monclient(hunting): picked mon.noname-a con 0x7f6e4c1402d0 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:09.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.709+0000 7f6e533cf640 10 monclient(hunting): picked mon.noname-b con 0x7f6e4c149ae0 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:09.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.709+0000 7f6e533cf640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4c13e730 con 0x7f6e4c1402d0 2024-03-25T18:29:09.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.709+0000 7f6e533cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4c134870 con 0x7f6e4c149ae0 2024-03-25T18:29:09.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.709+0000 7f6e533cf640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4c06daf0 con 0x7f6e4c0705b0 2024-03-25T18:29:09.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.709+0000 7f6e533cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:09.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.709+0000 7f6e533cf640 10 monclient(hunting): authenticate will time out at 7567.877769s 2024-03-25T18:29:09.729 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.728+0000 7f6e5196c640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f6e4c149ae0 0x7f6e4c134160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:42930/0 (socket says 172.21.15.99:42930) 2024-03-25T18:29:09.729 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.728+0000 7f6e5196c640 1 -- 172.21.15.99:0/4073789962 learned_addr learned my addr 172.21.15.99:0/4073789962 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:09.730 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.729+0000 7f6e41ffb640 10 client.?.objecter ms_handle_connect 0x7f6e4c149ae0 2024-03-25T18:29:09.730 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.729+0000 7f6e41ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:09.731 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.730+0000 7f6e41ffb640 10 client.?.objecter ms_handle_connect 0x7f6e4c0705b0 2024-03-25T18:29:09.731 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.730+0000 7f6e41ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:09.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.735+0000 7f6e41ffb640 10 client.?.objecter ms_handle_connect 0x7f6e4c1402d0 2024-03-25T18:29:09.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.735+0000 7f6e41ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:09.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.746+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328229110 0 0) 0x7f6e4c134870 con 0x7f6e4c149ae0 2024-03-25T18:29:09.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.746+0000 7f6e41ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:09.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.746+0000 7f6e41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:09.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.746+0000 7f6e41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:09.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.746+0000 7f6e41ffb640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:29:09.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.746+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e30004eb0 con 0x7f6e4c149ae0 2024-03-25T18:29:09.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.746+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120120580 0 0) 0x7f6e4c13e730 con 0x7f6e4c1402d0 2024-03-25T18:29:09.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.746+0000 7f6e41ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:09.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.746+0000 7f6e41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:09.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.746+0000 7f6e41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:09.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.746+0000 7f6e41ffb640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:29:09.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.746+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e4c134870 con 0x7f6e4c1402d0 2024-03-25T18:29:09.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.751+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372669253 0 0) 0x7f6e4c06daf0 con 0x7f6e4c0705b0 2024-03-25T18:29:09.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.751+0000 7f6e41ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:09.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.751+0000 7f6e41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:09.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.751+0000 7f6e41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:09.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.751+0000 7f6e41ffb640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:29:09.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.751+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e4c13e730 con 0x7f6e4c0705b0 2024-03-25T18:29:09.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.764+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2200032039 0 0) 0x7f6e4c134870 con 0x7f6e4c1402d0 2024-03-25T18:29:09.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.764+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e4c06daf0 con 0x7f6e4c1402d0 2024-03-25T18:29:09.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.764+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6e3c0018b0 con 0x7f6e4c1402d0 2024-03-25T18:29:09.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.764+0000 7f6e41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:09.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.764+0000 7f6e41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:09.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.764+0000 7f6e41ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:09.766 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:09.766 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:09.766 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:09.766 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:09.766 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:09.766 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:09.766 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:09.766 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:09.767 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:09.767 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:09.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.764+0000 7f6e41ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:09.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.767+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3150028353 0 0) 0x7f6e30004eb0 con 0x7f6e4c149ae0 2024-03-25T18:29:09.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.767+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e4c134870 con 0x7f6e4c149ae0 2024-03-25T18:29:09.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.767+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6e44001c30 con 0x7f6e4c149ae0 2024-03-25T18:29:09.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.767+0000 7f6e41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:09.769 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.767+0000 7f6e41ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:09.769 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.767+0000 7f6e41ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:09.769 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:09.769 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:09.769 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:09.769 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:09.769 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:09.769 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:09.770 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:09.770 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:09.770 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:09.770 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:09.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2560673281 0 0) 0x7f6e4c134870 con 0x7f6e4c149ae0 2024-03-25T18:29:09.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 >> v1:172.21.15.99:6790/0 conn(0x7f6e4c0705b0 legacy=0x7f6e4c1daa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:09.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 >> v1:172.21.15.99:6789/0 conn(0x7f6e4c1402d0 legacy=0x7f6e4c133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:09.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e41ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:09.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e41ffb640 1 monclient: found mon.b 2024-03-25T18:29:09.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e41ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c0af7e0 con 0x7f6e4c149ae0 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e41ffb640 10 monclient: _finish_auth 0 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e41ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e533cf640 5 monclient: authenticate success, global_id 4396 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e533cf640 10 monclient: _renew_subs 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e533cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e533cf640 1 -- 172.21.15.99:0/4073789962 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4c1db120 con 0x7f6e4c149ae0 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e533cf640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e533cf640 10 monclient: _renew_subs 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e533cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e533cf640 1 -- 172.21.15.99:0/4073789962 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4c136030 con 0x7f6e4c149ae0 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.781+0000 7f6e533cf640 1 librados: init done 2024-03-25T18:29:09.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.782+0000 7f6e533cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:09.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.782+0000 7f6e533cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:09.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.782+0000 7f6e533cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:09.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:09.782+0000 7f6e533cf640 1 -- 172.21.15.99:0/4073789962 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4c0627b0 con 0x7f6e4c149ae0 2024-03-25T18:29:10.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.051+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e44003970 con 0x7f6e4c149ae0 2024-03-25T18:29:10.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.051+0000 7f6e41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:10.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.051+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6e44002100 con 0x7f6e4c149ae0 2024-03-25T18:29:10.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.051+0000 7f6e41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:10.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.051+0000 7f6e41ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:10.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.051+0000 7f6e41ffb640 10 monclient: dump: 2024-03-25T18:29:10.053 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:10.053 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:10.053 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:10.053 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:10.053 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:10.053 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:10.053 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:10.053 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:10.053 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:10.053 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:10.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.051+0000 7f6e437fe640 4 set_mon_vals no callback set 2024-03-25T18:29:10.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.052+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f6e44003dc0 con 0x7f6e4c149ae0 2024-03-25T18:29:10.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.053+0000 7f6e5116b640 10 monclient: get_auth_request con 0x7f6e3006df60 auth_method 0 2024-03-25T18:29:10.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.053+0000 7f6e41ffb640 10 client.4396.objecter ms_handle_connect 0x7f6e3006df60 2024-03-25T18:29:10.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.059+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f6e44054a70 con 0x7f6e4c149ae0 2024-03-25T18:29:10.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.059+0000 7f6e41ffb640 10 client.4396.objecter ms_dispatch 0x7f6e4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:10.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.059+0000 7f6e41ffb640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:10.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.059+0000 7f6e41ffb640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:10.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.059+0000 7f6e41ffb640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T18:29:10.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.061+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e44050580 con 0x7f6e4c149ae0 2024-03-25T18:29:10.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.061+0000 7f6e41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:10.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.061+0000 7f6e41ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:10.312 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.311+0000 7f6e533cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:10.312 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.311+0000 7f6e533cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:10.313 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.311+0000 7f6e533cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:10.313 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.311+0000 7f6e533cf640 1 -- 172.21.15.99:0/4073789962 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e4c069f40 con 0x7f6e4c149ae0 2024-03-25T18:29:10.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.332+0000 7f6e41ffb640 1 -- 172.21.15.99:0/4073789962 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f6e440524b0 con 0x7f6e4c149ae0 2024-03-25T18:29:10.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.332+0000 7f6e41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:10.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.332+0000 7f6e41ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:10.333 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:29:10.333 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:10.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.334+0000 7f6e533cf640 10 librados: watch_flush enter 2024-03-25T18:29:10.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.334+0000 7f6e533cf640 10 librados: watch_flush exit 2024-03-25T18:29:10.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.334+0000 7f6e533cf640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T18:29:10.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.334+0000 7f6e533cf640 10 client.4396.objecter successfully canceled tick 2024-03-25T18:29:10.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.335+0000 7f6e533cf640 1 -- 172.21.15.99:0/4073789962 >> v1:172.21.15.190:6800/4220409699 conn(0x7f6e3006df60 legacy=0x7f6e30070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:10.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.335+0000 7f6e533cf640 10 monclient: shutdown 2024-03-25T18:29:10.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.335+0000 7f6e533cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:10.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.335+0000 7f6e533cf640 1 -- 172.21.15.99:0/4073789962 >> v1:172.21.15.190:6789/0 conn(0x7f6e4c149ae0 legacy=0x7f6e4c134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:10.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.335+0000 7f6e533cf640 1 -- 172.21.15.99:0/4073789962 shutdown_connections 2024-03-25T18:29:10.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.335+0000 7f6e533cf640 1 -- 172.21.15.99:0/4073789962 >> 172.21.15.99:0/4073789962 conn(0x7f6e4c0b5d10 msgr2=0x7f6e4c140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:10.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.335+0000 7f6e533cf640 1 -- 172.21.15.99:0/4073789962 shutdown_connections 2024-03-25T18:29:10.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.335+0000 7f6e533cf640 1 -- 172.21.15.99:0/4073789962 wait complete. 2024-03-25T18:29:10.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:10.335+0000 7f6e533cf640 1 librados: shutdown 2024-03-25T18:29:10.348 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:17.156 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:17.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.273+0000 7f3dce58c640 10 monclient: get_monmap_and_config 2024-03-25T18:29:17.276 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.273+0000 7f3dce58c640 10 monclient: build_initial_monmap 2024-03-25T18:29:17.276 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.273+0000 7f3dce58c640 10 monclient: monmap: 2024-03-25T18:29:17.276 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:17.276 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:17.276 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:29:17.274948+0000 2024-03-25T18:29:17.276 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:29:17.274948+0000 2024-03-25T18:29:17.276 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:17.276 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:17.276 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:17.276 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:29:17.276 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:29:17.276 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:17.277 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.274+0000 7f3dce58c640 1 Processor -- start 2024-03-25T18:29:17.277 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.274+0000 7f3dce58c640 1 -- start start 2024-03-25T18:29:17.277 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.274+0000 7f3dce58c640 10 monclient: init 2024-03-25T18:29:17.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.275+0000 7f3dce58c640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:17.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.275+0000 7f3dce58c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:17.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.275+0000 7f3dce58c640 10 monclient(hunting): picked mon.noname-c con 0x7f3dc80a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:17.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.275+0000 7f3dce58c640 10 monclient(hunting): picked mon.noname-a con 0x7f3dc813f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:17.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.275+0000 7f3dce58c640 10 monclient(hunting): picked mon.noname-b con 0x7f3dc81430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:17.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.275+0000 7f3dce58c640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc8142520 con 0x7f3dc813f710 2024-03-25T18:29:17.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.275+0000 7f3dce58c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc80b28a0 con 0x7f3dc80a98f0 2024-03-25T18:29:17.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.275+0000 7f3dce58c640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc80b2a30 con 0x7f3dc81430c0 2024-03-25T18:29:17.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.275+0000 7f3dce58c640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:17.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.275+0000 7f3dce58c640 10 monclient(hunting): authenticate will time out at 7575.443433s 2024-03-25T18:29:17.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.288+0000 7f3dc77fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f3dc813f710 0x7f3dc81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56498/0 (socket says 172.21.15.99:56498) 2024-03-25T18:29:17.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.288+0000 7f3dc77fe640 1 -- 172.21.15.99:0/1800528643 learned_addr learned my addr 172.21.15.99:0/1800528643 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:17.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.299+0000 7f3dc6ffd640 1 -- 172.21.15.99:0/1800528643 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557874986 0 0) 0x7f3dc8142520 con 0x7f3dc813f710 2024-03-25T18:29:17.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.299+0000 7f3dc6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:17.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.299+0000 7f3dc6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:17.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.299+0000 7f3dc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:17.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.299+0000 7f3dc6ffd640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:29:17.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.299+0000 7f3dc6ffd640 1 -- 172.21.15.99:0/1800528643 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db0003520 con 0x7f3dc813f710 2024-03-25T18:29:17.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.302+0000 7f3dc6ffd640 1 -- 172.21.15.99:0/1800528643 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269460720 0 0) 0x7f3dc80b28a0 con 0x7f3dc80a98f0 2024-03-25T18:29:17.304 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.302+0000 7f3dc6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:17.304 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.302+0000 7f3dc6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:17.304 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.302+0000 7f3dc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:17.304 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.302+0000 7f3dc6ffd640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:29:17.304 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.302+0000 7f3dc6ffd640 1 -- 172.21.15.99:0/1800528643 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dc8142520 con 0x7f3dc80a98f0 2024-03-25T18:29:17.315 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.314+0000 7f3dc6ffd640 1 -- 172.21.15.99:0/1800528643 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575680692 0 0) 0x7f3dc80b2a30 con 0x7f3dc81430c0 2024-03-25T18:29:17.316 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.315+0000 7f3dc6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:17.316 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.315+0000 7f3dc6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:17.316 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.315+0000 7f3dc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:17.316 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.315+0000 7f3dc6ffd640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:29:17.316 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.315+0000 7f3dc6ffd640 1 -- 172.21.15.99:0/1800528643 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dc80b28a0 con 0x7f3dc81430c0 2024-03-25T18:29:17.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 1 -- 172.21.15.99:0/1800528643 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1532880201 0 0) 0x7f3db0003520 con 0x7f3dc813f710 2024-03-25T18:29:17.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 1 -- 172.21.15.99:0/1800528643 >> v1:172.21.15.99:6790/0 conn(0x7f3dc81430c0 legacy=0x7f3dc814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:17.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 1 -- 172.21.15.99:0/1800528643 >> v1:172.21.15.190:6789/0 conn(0x7f3dc80a98f0 legacy=0x7f3dc80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:17.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:17.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 1 monclient: found mon.noname-a 2024-03-25T18:29:17.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:17.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:29:17.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 1 -- 172.21.15.99:0/1800528643 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc80b44c0 con 0x7f3dc813f710 2024-03-25T18:29:17.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:17.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:17.320 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:17.320 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 1 -- 172.21.15.99:0/1800528643 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f3dbc001690 con 0x7f3dc813f710 2024-03-25T18:29:17.320 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dce58c640 5 monclient: authenticate success, global_id 4383 2024-03-25T18:29:17.320 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dce58c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:17.320 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:17.320 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:17.320 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dc6ffd640 10 monclient: dump: 2024-03-25T18:29:17.320 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:17.320 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:17.320 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:17.320 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:17.320 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:17.320 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:17.321 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:17.321 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:17.321 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:17.321 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:17.321 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.317+0000 7f3dce58c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:17.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.336+0000 7f3dc6ffd640 1 -- 172.21.15.99:0/1800528643 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3dbc0017f0 con 0x7f3dc813f710 2024-03-25T18:29:17.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.336+0000 7f3dc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:17.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.336+0000 7f3dc6ffd640 1 -- 172.21.15.99:0/1800528643 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f3dbc001c00 con 0x7f3dc813f710 2024-03-25T18:29:17.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.336+0000 7f3dc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:17.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.336+0000 7f3dc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:17.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.336+0000 7f3dc6ffd640 10 monclient: dump: 2024-03-25T18:29:17.338 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:17.338 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:17.338 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:17.338 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:17.338 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:17.338 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:17.339 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:17.339 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:17.339 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:17.339 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:17.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.336+0000 7f3dce58c640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:17.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.336+0000 7f3dce58c640 4 set_mon_vals no callback set 2024-03-25T18:29:17.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.337+0000 7f3dce58c640 10 monclient: shutdown 2024-03-25T18:29:17.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.337+0000 7f3dce58c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:17.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.337+0000 7f3dce58c640 1 -- 172.21.15.99:0/1800528643 >> v1:172.21.15.99:6789/0 conn(0x7f3dc813f710 legacy=0x7f3dc81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:17.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.337+0000 7f3dce58c640 1 -- 172.21.15.99:0/1800528643 shutdown_connections 2024-03-25T18:29:17.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.337+0000 7f3dce58c640 1 -- 172.21.15.99:0/1800528643 >> 172.21.15.99:0/1800528643 conn(0x7f3dc80b7ea0 msgr2=0x7f3dc80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:17.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.337+0000 7f3dce58c640 1 -- 172.21.15.99:0/1800528643 shutdown_connections 2024-03-25T18:29:17.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.337+0000 7f3dce58c640 1 -- 172.21.15.99:0/1800528643 wait complete. 2024-03-25T18:29:17.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.338+0000 7f3dce58c640 10 monclient: build_initial_monmap 2024-03-25T18:29:17.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.338+0000 7f3dce58c640 10 monclient: monmap: 2024-03-25T18:29:17.342 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:17.342 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:17.342 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:29:17.342 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:29:17.343 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:17.343 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:29:17.343 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:17.343 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:29:17.343 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:29:17.343 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:17.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.338+0000 7f3dce58c640 1 librados: starting msgr at 2024-03-25T18:29:17.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.338+0000 7f3dce58c640 1 librados: starting objecter 2024-03-25T18:29:17.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.338+0000 7f3dce58c640 1 Processor -- start 2024-03-25T18:29:17.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.339+0000 7f3dce58c640 1 -- start start 2024-03-25T18:29:17.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.339+0000 7f3dce58c640 1 librados: setting wanted keys 2024-03-25T18:29:17.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.339+0000 7f3dce58c640 1 librados: calling monclient init 2024-03-25T18:29:17.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.339+0000 7f3dce58c640 10 monclient: init 2024-03-25T18:29:17.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.339+0000 7f3dce58c640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:17.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.339+0000 7f3dce58c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:17.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.339+0000 7f3dce58c640 10 monclient(hunting): picked mon.noname-a con 0x7f3dc80a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:17.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.339+0000 7f3dce58c640 10 monclient(hunting): picked mon.noname-b con 0x7f3dc813f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:17.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.339+0000 7f3dce58c640 10 monclient(hunting): picked mon.noname-c con 0x7f3dc81430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:17.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.339+0000 7f3dce58c640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc80abac0 con 0x7f3dc80a98f0 2024-03-25T18:29:17.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.339+0000 7f3dce58c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc80abda0 con 0x7f3dc813f710 2024-03-25T18:29:17.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.339+0000 7f3dce58c640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc81380e0 con 0x7f3dc81430c0 2024-03-25T18:29:17.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.339+0000 7f3dce58c640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:17.344 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.339+0000 7f3dce58c640 10 monclient(hunting): authenticate will time out at 7575.507681s 2024-03-25T18:29:17.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.349+0000 7f3dc7fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f3dc80a98f0 0x7f3dc80ac580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56560/0 (socket says 172.21.15.99:56560) 2024-03-25T18:29:17.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.349+0000 7f3dc7fff640 1 -- 172.21.15.99:0/2466567500 learned_addr learned my addr 172.21.15.99:0/2466567500 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:17.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.352+0000 7f3dc4ff9640 10 client.?.objecter ms_handle_connect 0x7f3dc80a98f0 2024-03-25T18:29:17.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.352+0000 7f3dc4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:17.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.353+0000 7f3dc4ff9640 10 client.?.objecter ms_handle_connect 0x7f3dc813f710 2024-03-25T18:29:17.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.353+0000 7f3dc4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:17.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.357+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420308403 0 0) 0x7f3dc80abac0 con 0x7f3dc80a98f0 2024-03-25T18:29:17.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.357+0000 7f3dc4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:17.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.357+0000 7f3dc4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:17.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.357+0000 7f3dc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:17.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.357+0000 7f3dc4ff9640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:29:17.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.357+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db4004eb0 con 0x7f3dc80a98f0 2024-03-25T18:29:17.366 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.365+0000 7f3dc4ff9640 10 client.?.objecter ms_handle_connect 0x7f3dc81430c0 2024-03-25T18:29:17.366 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.365+0000 7f3dc4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:17.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.366+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3506103206 0 0) 0x7f3db4004eb0 con 0x7f3dc80a98f0 2024-03-25T18:29:17.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.366+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3dc80abac0 con 0x7f3dc80a98f0 2024-03-25T18:29:17.370 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.369+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f3db8001a90 con 0x7f3dc80a98f0 2024-03-25T18:29:17.370 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.369+0000 7f3dc4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:17.370 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.369+0000 7f3dc4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:17.370 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.369+0000 7f3dc4ff9640 10 monclient(hunting): dump: 2024-03-25T18:29:17.370 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:17.370 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:17.370 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:17.371 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:17.371 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:17.371 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:17.371 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:17.371 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:17.371 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:17.371 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:17.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.369+0000 7f3dc4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:17.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.372+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651985316 0 0) 0x7f3dc80abda0 con 0x7f3dc813f710 2024-03-25T18:29:17.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.372+0000 7f3dc4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:17.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.372+0000 7f3dc4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:17.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.372+0000 7f3dc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:17.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.372+0000 7f3dc4ff9640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:29:17.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.372+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db4004eb0 con 0x7f3dc813f710 2024-03-25T18:29:17.380 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.379+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953269606 0 0) 0x7f3dc81380e0 con 0x7f3dc81430c0 2024-03-25T18:29:17.380 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.379+0000 7f3dc4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:17.380 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.379+0000 7f3dc4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:17.380 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.379+0000 7f3dc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:17.380 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.379+0000 7f3dc4ff9640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:29:17.380 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.379+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dc80abda0 con 0x7f3dc81430c0 2024-03-25T18:29:17.387 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3421323003 0 0) 0x7f3dc80abac0 con 0x7f3dc80a98f0 2024-03-25T18:29:17.387 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 >> v1:172.21.15.99:6790/0 conn(0x7f3dc81430c0 legacy=0x7f3dc81e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:17.387 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 >> v1:172.21.15.190:6789/0 conn(0x7f3dc813f710 legacy=0x7f3dc8138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:17.387 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dc4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:29:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dc4ff9640 1 monclient: found mon.a 2024-03-25T18:29:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dc4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dc4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc8137d90 con 0x7f3dc80a98f0 2024-03-25T18:29:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dc4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:29:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dc4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:29:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dc4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dce58c640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:29:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dce58c640 10 monclient: _renew_subs 2024-03-25T18:29:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dce58c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dce58c640 1 -- 172.21.15.99:0/2466567500 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc8139240 con 0x7f3dc80a98f0 2024-03-25T18:29:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dce58c640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dce58c640 10 monclient: _renew_subs 2024-03-25T18:29:17.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dce58c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:17.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dce58c640 1 -- 172.21.15.99:0/2466567500 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc8136830 con 0x7f3dc80a98f0 2024-03-25T18:29:17.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.386+0000 7f3dce58c640 1 librados: init done 2024-03-25T18:29:17.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.387+0000 7f3dce58c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:17.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.387+0000 7f3dce58c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:17.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.387+0000 7f3dce58c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:17.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.387+0000 7f3dce58c640 1 -- 172.21.15.99:0/2466567500 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc80627b0 con 0x7f3dc80a98f0 2024-03-25T18:29:17.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.684+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3db8001bf0 con 0x7f3dc80a98f0 2024-03-25T18:29:17.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.684+0000 7f3dc4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:17.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.684+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f3db8003a60 con 0x7f3dc80a98f0 2024-03-25T18:29:17.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.684+0000 7f3dc4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:17.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.684+0000 7f3dc4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:17.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.684+0000 7f3dc4ff9640 10 monclient: dump: 2024-03-25T18:29:17.686 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:17.686 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:17.686 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:17.686 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:17.686 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:17.686 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:17.686 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:17.686 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:17.686 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:17.686 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:17.687 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.684+0000 7f3dc67fc640 4 set_mon_vals no callback set 2024-03-25T18:29:17.687 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.685+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f3db8003c80 con 0x7f3dc80a98f0 2024-03-25T18:29:17.687 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.685+0000 7f3dc77fe640 10 monclient: get_auth_request con 0x7f3db406de00 auth_method 0 2024-03-25T18:29:17.687 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.686+0000 7f3dc4ff9640 10 client.4386.objecter ms_handle_connect 0x7f3db406de00 2024-03-25T18:29:17.692 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.691+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f3db8054860 con 0x7f3dc80a98f0 2024-03-25T18:29:17.692 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.691+0000 7f3dc4ff9640 10 client.4386.objecter ms_dispatch 0x7f3dc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:17.692 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.691+0000 7f3dc4ff9640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:17.692 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.691+0000 7f3dc4ff9640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:17.692 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.691+0000 7f3dc4ff9640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-25T18:29:17.692 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.691+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3db8050370 con 0x7f3dc80a98f0 2024-03-25T18:29:17.693 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.691+0000 7f3dc4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:17.693 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.691+0000 7f3dc4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:17.947 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.946+0000 7f3dce58c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:17.947 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.946+0000 7f3dce58c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:17.947 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.946+0000 7f3dce58c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:17.947 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:17.946+0000 7f3dce58c640 1 -- 172.21.15.99:0/2466567500 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3dc81393d0 con 0x7f3dc80a98f0 2024-03-25T18:29:18.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.026+0000 7f3dc4ff9640 1 -- 172.21.15.99:0/2466567500 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f3db80522a0 con 0x7f3dc80a98f0 2024-03-25T18:29:18.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.026+0000 7f3dc4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:18.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.026+0000 7f3dc4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:18.028 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:29:18.028 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:18.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.028+0000 7f3dce58c640 10 librados: watch_flush enter 2024-03-25T18:29:18.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.028+0000 7f3dce58c640 10 librados: watch_flush exit 2024-03-25T18:29:18.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.028+0000 7f3dce58c640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-25T18:29:18.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.028+0000 7f3dce58c640 10 client.4386.objecter successfully canceled tick 2024-03-25T18:29:18.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.028+0000 7f3dce58c640 1 -- 172.21.15.99:0/2466567500 >> v1:172.21.15.190:6800/4220409699 conn(0x7f3db406de00 legacy=0x7f3db40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:18.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.028+0000 7f3dce58c640 10 monclient: shutdown 2024-03-25T18:29:18.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.028+0000 7f3dce58c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:18.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.028+0000 7f3dce58c640 1 -- 172.21.15.99:0/2466567500 >> v1:172.21.15.99:6789/0 conn(0x7f3dc80a98f0 legacy=0x7f3dc80ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:18.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.029+0000 7f3dce58c640 1 -- 172.21.15.99:0/2466567500 shutdown_connections 2024-03-25T18:29:18.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.029+0000 7f3dce58c640 1 -- 172.21.15.99:0/2466567500 >> 172.21.15.99:0/2466567500 conn(0x7f3dc80b7ea0 msgr2=0x7f3dc8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:18.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.029+0000 7f3dce58c640 1 -- 172.21.15.99:0/2466567500 shutdown_connections 2024-03-25T18:29:18.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.029+0000 7f3dce58c640 1 -- 172.21.15.99:0/2466567500 wait complete. 2024-03-25T18:29:18.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:18.029+0000 7f3dce58c640 1 librados: shutdown 2024-03-25T18:29:18.042 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:24.850 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:24.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.961+0000 7f63b91e9640 10 monclient: get_monmap_and_config 2024-03-25T18:29:24.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.961+0000 7f63b91e9640 10 monclient: build_initial_monmap 2024-03-25T18:29:24.963 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.961+0000 7f63b91e9640 10 monclient: monmap: 2024-03-25T18:29:24.963 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:24.963 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:24.963 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:29:24.962728+0000 2024-03-25T18:29:24.963 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:29:24.962728+0000 2024-03-25T18:29:24.963 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:24.963 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:24.964 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:24.964 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:29:24.964 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:29:24.964 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:24.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.962+0000 7f63b91e9640 1 Processor -- start 2024-03-25T18:29:24.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.962+0000 7f63b91e9640 1 -- start start 2024-03-25T18:29:24.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.962+0000 7f63b91e9640 10 monclient: init 2024-03-25T18:29:24.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.963+0000 7f63b91e9640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:24.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.963+0000 7f63b91e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:24.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.963+0000 7f63b91e9640 10 monclient(hunting): picked mon.noname-b con 0x7f63b40a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:24.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.963+0000 7f63b91e9640 10 monclient(hunting): picked mon.noname-c con 0x7f63b413f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:24.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.963+0000 7f63b91e9640 10 monclient(hunting): picked mon.noname-a con 0x7f63b41430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:24.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.963+0000 7f63b91e9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b4142520 con 0x7f63b41430c0 2024-03-25T18:29:24.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.963+0000 7f63b91e9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b40b28a0 con 0x7f63b413f710 2024-03-25T18:29:24.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.963+0000 7f63b91e9640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b40b2a30 con 0x7f63b40a98f0 2024-03-25T18:29:24.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.963+0000 7f63b91e9640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:24.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.963+0000 7f63b91e9640 10 monclient(hunting): authenticate will time out at 7583.131230s 2024-03-25T18:29:24.971 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.970+0000 7f63b359e640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f63b41430c0 0x7f63b414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:44844/0 (socket says 172.21.15.99:44844) 2024-03-25T18:29:24.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.970+0000 7f63b359e640 1 -- 172.21.15.99:0/668745045 learned_addr learned my addr 172.21.15.99:0/668745045 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:24.984 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.983+0000 7f63b259c640 1 -- 172.21.15.99:0/668745045 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855231258 0 0) 0x7f63b4142520 con 0x7f63b41430c0 2024-03-25T18:29:24.984 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.983+0000 7f63b259c640 10 monclient(hunting): _negotiate 2024-03-25T18:29:24.985 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.983+0000 7f63b259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:24.985 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.983+0000 7f63b259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:24.985 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.983+0000 7f63b259c640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:29:24.985 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.983+0000 7f63b259c640 1 -- 172.21.15.99:0/668745045 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a0003520 con 0x7f63b41430c0 2024-03-25T18:29:24.987 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.986+0000 7f63b259c640 1 -- 172.21.15.99:0/668745045 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648168368 0 0) 0x7f63b40b2a30 con 0x7f63b40a98f0 2024-03-25T18:29:24.987 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.986+0000 7f63b259c640 10 monclient(hunting): _negotiate 2024-03-25T18:29:24.987 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.986+0000 7f63b259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:24.987 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.986+0000 7f63b259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:24.987 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.986+0000 7f63b259c640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:29:24.987 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.986+0000 7f63b259c640 1 -- 172.21.15.99:0/668745045 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b4142520 con 0x7f63b40a98f0 2024-03-25T18:29:24.992 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.991+0000 7f63b259c640 1 -- 172.21.15.99:0/668745045 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347805121 0 0) 0x7f63b40b28a0 con 0x7f63b413f710 2024-03-25T18:29:24.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.991+0000 7f63b259c640 10 monclient(hunting): _negotiate 2024-03-25T18:29:24.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.991+0000 7f63b259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:24.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.991+0000 7f63b259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:24.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.991+0000 7f63b259c640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:29:24.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.991+0000 7f63b259c640 1 -- 172.21.15.99:0/668745045 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b40b2a30 con 0x7f63b413f710 2024-03-25T18:29:25.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 1 -- 172.21.15.99:0/668745045 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 74048714 0 0) 0x7f63a0003520 con 0x7f63b41430c0 2024-03-25T18:29:25.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 1 -- 172.21.15.99:0/668745045 >> v1:172.21.15.99:6790/0 conn(0x7f63b40a98f0 legacy=0x7f63b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 1 -- 172.21.15.99:0/668745045 >> v1:172.21.15.190:6789/0 conn(0x7f63b413f710 legacy=0x7f63b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 10 monclient: _finish_hunting 0 2024-03-25T18:29:25.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 1 monclient: found mon.noname-a 2024-03-25T18:29:25.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:25.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:29:25.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 1 -- 172.21.15.99:0/668745045 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b40b44c0 con 0x7f63b41430c0 2024-03-25T18:29:25.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 10 monclient: _finish_auth 0 2024-03-25T18:29:25.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 10 monclient: _check_auth_tickets 2024-03-25T18:29:25.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:25.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 1 -- 172.21.15.99:0/668745045 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f63ac001a10 con 0x7f63b41430c0 2024-03-25T18:29:25.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b91e9640 5 monclient: authenticate success, global_id 4389 2024-03-25T18:29:25.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b91e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:25.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:25.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:25.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b259c640 10 monclient: dump: 2024-03-25T18:29:25.001 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:25.001 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:25.002 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:25.002 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:25.002 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:25.002 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:25.002 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:25.002 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:25.002 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:25.002 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:25.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:24.999+0000 7f63b91e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:25.007 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.006+0000 7f63b259c640 1 -- 172.21.15.99:0/668745045 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63ac001b70 con 0x7f63b41430c0 2024-03-25T18:29:25.007 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.006+0000 7f63b259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:25.008 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.006+0000 7f63b259c640 1 -- 172.21.15.99:0/668745045 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f63ac001f80 con 0x7f63b41430c0 2024-03-25T18:29:25.008 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.006+0000 7f63b259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:25.008 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.006+0000 7f63b259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:25.008 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.006+0000 7f63b259c640 10 monclient: dump: 2024-03-25T18:29:25.008 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:25.008 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:25.008 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:25.008 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:25.008 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:25.008 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:25.008 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:25.008 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:25.008 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:25.008 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:25.009 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.006+0000 7f63b91e9640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:25.009 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.006+0000 7f63b91e9640 4 set_mon_vals no callback set 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.007+0000 7f63b91e9640 10 monclient: shutdown 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.007+0000 7f63b91e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.007+0000 7f63b91e9640 1 -- 172.21.15.99:0/668745045 >> v1:172.21.15.99:6789/0 conn(0x7f63b41430c0 legacy=0x7f63b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.007+0000 7f63b91e9640 1 -- 172.21.15.99:0/668745045 shutdown_connections 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.007+0000 7f63b91e9640 1 -- 172.21.15.99:0/668745045 >> 172.21.15.99:0/668745045 conn(0x7f63b40b7ea0 msgr2=0x7f63b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.007+0000 7f63b91e9640 1 -- 172.21.15.99:0/668745045 shutdown_connections 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.007+0000 7f63b91e9640 1 -- 172.21.15.99:0/668745045 wait complete. 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.008+0000 7f63b91e9640 10 monclient: build_initial_monmap 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.008+0000 7f63b91e9640 10 monclient: monmap: 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:25.011 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:29:25.012 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:29:25.012 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:25.012 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:29:25.012 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:25.012 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:29:25.012 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:29:25.012 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:25.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.008+0000 7f63b91e9640 1 librados: starting msgr at 2024-03-25T18:29:25.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.008+0000 7f63b91e9640 1 librados: starting objecter 2024-03-25T18:29:25.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.008+0000 7f63b91e9640 1 Processor -- start 2024-03-25T18:29:25.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.008+0000 7f63b91e9640 1 -- start start 2024-03-25T18:29:25.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.008+0000 7f63b91e9640 1 librados: setting wanted keys 2024-03-25T18:29:25.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.008+0000 7f63b91e9640 1 librados: calling monclient init 2024-03-25T18:29:25.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.008+0000 7f63b91e9640 10 monclient: init 2024-03-25T18:29:25.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.009+0000 7f63b91e9640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:25.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.009+0000 7f63b91e9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:25.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.009+0000 7f63b91e9640 10 monclient(hunting): picked mon.noname-a con 0x7f63b40a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:25.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.009+0000 7f63b91e9640 10 monclient(hunting): picked mon.noname-c con 0x7f63b413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:25.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.009+0000 7f63b91e9640 10 monclient(hunting): picked mon.noname-b con 0x7f63b41430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:25.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.009+0000 7f63b91e9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b4139830 con 0x7f63b40a98f0 2024-03-25T18:29:25.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.009+0000 7f63b91e9640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b41db7d0 con 0x7f63b41430c0 2024-03-25T18:29:25.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.009+0000 7f63b91e9640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b41db410 con 0x7f63b413f710 2024-03-25T18:29:25.014 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.009+0000 7f63b91e9640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:25.014 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.009+0000 7f63b91e9640 10 monclient(hunting): authenticate will time out at 7583.177278s 2024-03-25T18:29:25.014 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.011+0000 7f63abfff640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f63b413f710 0x7f63b40b0220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:56872/0 (socket says 172.21.15.99:56872) 2024-03-25T18:29:25.014 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.011+0000 7f63abfff640 1 -- 172.21.15.99:0/1320879585 learned_addr learned my addr 172.21.15.99:0/1320879585 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:25.014 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.012+0000 7f63ab7fe640 10 client.?.objecter ms_handle_connect 0x7f63b413f710 2024-03-25T18:29:25.015 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.012+0000 7f63ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:25.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.020+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077949841 0 0) 0x7f63b41db410 con 0x7f63b413f710 2024-03-25T18:29:25.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.020+0000 7f63ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:25.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.020+0000 7f63ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:25.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.020+0000 7f63ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:25.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.020+0000 7f63ab7fe640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:29:25.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.020+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6394004eb0 con 0x7f63b413f710 2024-03-25T18:29:25.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.035+0000 7f63ab7fe640 10 client.?.objecter ms_handle_connect 0x7f63b41430c0 2024-03-25T18:29:25.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.035+0000 7f63ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:25.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.036+0000 7f63ab7fe640 10 client.?.objecter ms_handle_connect 0x7f63b40a98f0 2024-03-25T18:29:25.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.036+0000 7f63ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:25.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.037+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2007896903 0 0) 0x7f6394004eb0 con 0x7f63b413f710 2024-03-25T18:29:25.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.037+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63b41db410 con 0x7f63b413f710 2024-03-25T18:29:25.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.037+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f639c001a30 con 0x7f63b413f710 2024-03-25T18:29:25.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.037+0000 7f63ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:25.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.037+0000 7f63ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:25.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.037+0000 7f63ab7fe640 10 monclient(hunting): dump: 2024-03-25T18:29:25.039 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:25.039 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:25.039 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:25.039 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:25.039 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:25.039 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:25.040 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:25.040 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:25.040 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:25.040 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:25.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.037+0000 7f63ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:25.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.042+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739272737 0 0) 0x7f63b4139830 con 0x7f63b40a98f0 2024-03-25T18:29:25.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.042+0000 7f63ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:25.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.042+0000 7f63ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:25.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.042+0000 7f63ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:25.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.042+0000 7f63ab7fe640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:29:25.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.042+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6394004eb0 con 0x7f63b40a98f0 2024-03-25T18:29:25.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.048+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271472514 0 0) 0x7f63b41db7d0 con 0x7f63b41430c0 2024-03-25T18:29:25.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.048+0000 7f63ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:25.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.048+0000 7f63ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:25.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.048+0000 7f63ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:25.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.048+0000 7f63ab7fe640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:29:25.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.048+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b4139830 con 0x7f63b41430c0 2024-03-25T18:29:25.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 56835942 0 0) 0x7f63b41db410 con 0x7f63b413f710 2024-03-25T18:29:25.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 >> v1:172.21.15.190:6789/0 conn(0x7f63b41430c0 legacy=0x7f63b41d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 >> v1:172.21.15.99:6789/0 conn(0x7f63b40a98f0 legacy=0x7f63b41dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.051 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:25.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63ab7fe640 1 monclient: found mon.c 2024-03-25T18:29:25.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:25.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63ab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:29:25.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b41db5a0 con 0x7f63b413f710 2024-03-25T18:29:25.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63ab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:25.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:25.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:25.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63b91e9640 5 monclient: authenticate success, global_id 4400 2024-03-25T18:29:25.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63b91e9640 10 monclient: _renew_subs 2024-03-25T18:29:25.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63b91e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:29:25.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63b91e9640 1 -- 172.21.15.99:0/1320879585 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b40b1480 con 0x7f63b413f710 2024-03-25T18:29:25.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63b91e9640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:25.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63b91e9640 10 monclient: _renew_subs 2024-03-25T18:29:25.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63b91e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:29:25.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63b91e9640 1 -- 172.21.15.99:0/1320879585 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b40b0b10 con 0x7f63b413f710 2024-03-25T18:29:25.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.049+0000 7f63b91e9640 1 librados: init done 2024-03-25T18:29:25.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.050+0000 7f63b91e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:25.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.050+0000 7f63b91e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:25.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.050+0000 7f63b91e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:29:25.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.050+0000 7f63b91e9640 1 -- 172.21.15.99:0/1320879585 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b41db960 con 0x7f63b413f710 2024-03-25T18:29:25.325 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.325+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f639c001cb0 con 0x7f63b413f710 2024-03-25T18:29:25.326 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.325+0000 7f63ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:25.326 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.325+0000 7f63b1d9b640 4 set_mon_vals no callback set 2024-03-25T18:29:25.335 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.335+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f639c003a60 con 0x7f63b413f710 2024-03-25T18:29:25.335 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.335+0000 7f63ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:25.335 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.335+0000 7f63ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:25.335 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.335+0000 7f63ab7fe640 10 monclient: dump: 2024-03-25T18:29:25.335 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:25.336 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:25.336 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:25.336 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:25.336 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:25.336 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:25.336 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:25.336 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:25.336 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:25.336 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:25.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.336+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f639c003c80 con 0x7f63b413f710 2024-03-25T18:29:25.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.337+0000 7f63b2d9d640 10 monclient: get_auth_request con 0x7f639406de10 auth_method 0 2024-03-25T18:29:25.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.337+0000 7f63ab7fe640 10 client.4400.objecter ms_handle_connect 0x7f639406de10 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.341+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f639c054960 con 0x7f63b413f710 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.341+0000 7f63ab7fe640 10 client.4400.objecter ms_dispatch 0x7f63b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.342+0000 7f63ab7fe640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.342+0000 7f63ab7fe640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.342+0000 7f63ab7fe640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.342+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f639c0502a0 con 0x7f63b413f710 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.342+0000 7f63ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:25.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.342+0000 7f63ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:25.586 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.585+0000 7f63b91e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:25.586 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.585+0000 7f63b91e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:25.587 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.585+0000 7f63b91e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:29:25.587 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.585+0000 7f63b91e9640 1 -- 172.21.15.99:0/1320879585 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63b40627b0 con 0x7f63b413f710 2024-03-25T18:29:25.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.613+0000 7f63ab7fe640 1 -- 172.21.15.99:0/1320879585 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f639c0523a0 con 0x7f63b413f710 2024-03-25T18:29:25.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.613+0000 7f63ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:25.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.613+0000 7f63ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:25.615 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:29:25.615 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:25.617 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.616+0000 7f63b91e9640 10 librados: watch_flush enter 2024-03-25T18:29:25.617 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.616+0000 7f63b91e9640 10 librados: watch_flush exit 2024-03-25T18:29:25.618 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.616+0000 7f63b91e9640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-25T18:29:25.618 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.616+0000 7f63b91e9640 10 client.4400.objecter successfully canceled tick 2024-03-25T18:29:25.618 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.616+0000 7f63b91e9640 1 -- 172.21.15.99:0/1320879585 >> v1:172.21.15.190:6800/4220409699 conn(0x7f639406de10 legacy=0x7f63940702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.618 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.616+0000 7f63b91e9640 10 monclient: shutdown 2024-03-25T18:29:25.618 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.616+0000 7f63b91e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:25.618 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.616+0000 7f63b91e9640 1 -- 172.21.15.99:0/1320879585 >> v1:172.21.15.99:6790/0 conn(0x7f63b413f710 legacy=0x7f63b40b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.618 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.616+0000 7f63b91e9640 1 -- 172.21.15.99:0/1320879585 shutdown_connections 2024-03-25T18:29:25.618 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.616+0000 7f63b91e9640 1 -- 172.21.15.99:0/1320879585 >> 172.21.15.99:0/1320879585 conn(0x7f63b40b7ea0 msgr2=0x7f63b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:25.618 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.616+0000 7f63b91e9640 1 -- 172.21.15.99:0/1320879585 shutdown_connections 2024-03-25T18:29:25.618 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.616+0000 7f63b91e9640 1 -- 172.21.15.99:0/1320879585 wait complete. 2024-03-25T18:29:25.618 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:25.616+0000 7f63b91e9640 1 librados: shutdown 2024-03-25T18:29:25.629 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:32.436 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:32.551 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.549+0000 7f236c3e8640 10 monclient: get_monmap_and_config 2024-03-25T18:29:32.551 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.549+0000 7f236c3e8640 10 monclient: build_initial_monmap 2024-03-25T18:29:32.551 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.549+0000 7f236c3e8640 10 monclient: monmap: 2024-03-25T18:29:32.551 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:32.551 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:32.551 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:29:32.550689+0000 2024-03-25T18:29:32.551 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:29:32.550689+0000 2024-03-25T18:29:32.551 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:32.551 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:32.551 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:32.551 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:29:32.552 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:29:32.552 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:32.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.550+0000 7f236c3e8640 1 Processor -- start 2024-03-25T18:29:32.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.550+0000 7f236c3e8640 1 -- start start 2024-03-25T18:29:32.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.550+0000 7f236c3e8640 10 monclient: init 2024-03-25T18:29:32.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.551+0000 7f236c3e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:32.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.551+0000 7f236c3e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:32.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.551+0000 7f236c3e8640 10 monclient(hunting): picked mon.noname-a con 0x7f23640a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:32.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.551+0000 7f236c3e8640 10 monclient(hunting): picked mon.noname-b con 0x7f236413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:32.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.551+0000 7f236c3e8640 10 monclient(hunting): picked mon.noname-c con 0x7f23641430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:32.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.551+0000 7f236c3e8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2364142520 con 0x7f23640a98f0 2024-03-25T18:29:32.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.551+0000 7f236c3e8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23640b28a0 con 0x7f23641430c0 2024-03-25T18:29:32.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.551+0000 7f236c3e8640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23640b2a30 con 0x7f236413f710 2024-03-25T18:29:32.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.551+0000 7f236c3e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:32.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.551+0000 7f236c3e8640 10 monclient(hunting): authenticate will time out at 7590.719522s 2024-03-25T18:29:32.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.559+0000 7f236a985640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f23641430c0 0x7f236414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:51150/0 (socket says 172.21.15.99:51150) 2024-03-25T18:29:32.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.559+0000 7f236a985640 1 -- 172.21.15.99:0/303944418 learned_addr learned my addr 172.21.15.99:0/303944418 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:32.568 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.567+0000 7f2369182640 1 -- 172.21.15.99:0/303944418 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521059465 0 0) 0x7f23640b28a0 con 0x7f23641430c0 2024-03-25T18:29:32.568 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.567+0000 7f2369182640 10 monclient(hunting): _negotiate 2024-03-25T18:29:32.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.567+0000 7f2369182640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:32.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.567+0000 7f2369182640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:32.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.567+0000 7f2369182640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:29:32.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.567+0000 7f2369182640 1 -- 172.21.15.99:0/303944418 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2350003520 con 0x7f23641430c0 2024-03-25T18:29:32.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.576+0000 7f2369182640 1 -- 172.21.15.99:0/303944418 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741783642 0 0) 0x7f2364142520 con 0x7f23640a98f0 2024-03-25T18:29:32.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.576+0000 7f2369182640 10 monclient(hunting): _negotiate 2024-03-25T18:29:32.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.576+0000 7f2369182640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:32.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.576+0000 7f2369182640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:32.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.576+0000 7f2369182640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:29:32.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.576+0000 7f2369182640 1 -- 172.21.15.99:0/303944418 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23640b28a0 con 0x7f23640a98f0 2024-03-25T18:29:32.582 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 1 -- 172.21.15.99:0/303944418 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3571034174 0 0) 0x7f2350003520 con 0x7f23641430c0 2024-03-25T18:29:32.582 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 1 -- 172.21.15.99:0/303944418 >> v1:172.21.15.99:6790/0 conn(0x7f236413f710 legacy=0x7f23641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:32.582 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 1 -- 172.21.15.99:0/303944418 >> v1:172.21.15.99:6789/0 conn(0x7f23640a98f0 legacy=0x7f23640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:32.582 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 10 monclient: _finish_hunting 0 2024-03-25T18:29:32.582 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 1 monclient: found mon.noname-c 2024-03-25T18:29:32.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:32.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:29:32.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 1 -- 172.21.15.99:0/303944418 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23640b44c0 con 0x7f23641430c0 2024-03-25T18:29:32.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 10 monclient: _finish_auth 0 2024-03-25T18:29:32.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 10 monclient: _check_auth_tickets 2024-03-25T18:29:32.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:32.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 1 -- 172.21.15.99:0/303944418 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f235c001690 con 0x7f23641430c0 2024-03-25T18:29:32.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:32.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:32.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f2369182640 10 monclient: dump: 2024-03-25T18:29:32.583 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:32.583 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:32.583 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:32.584 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:32.584 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:32.584 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:32.584 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:32.584 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:32.584 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:32.584 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:32.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f236c3e8640 5 monclient: authenticate success, global_id 4411 2024-03-25T18:29:32.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.581+0000 7f236c3e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:32.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.596+0000 7f2369182640 1 -- 172.21.15.99:0/303944418 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f235c0017f0 con 0x7f23641430c0 2024-03-25T18:29:32.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.596+0000 7f2369182640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:32.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.596+0000 7f2369182640 1 -- 172.21.15.99:0/303944418 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f235c001c00 con 0x7f23641430c0 2024-03-25T18:29:32.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.596+0000 7f2369182640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:32.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.596+0000 7f2369182640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:32.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.596+0000 7f2369182640 10 monclient: dump: 2024-03-25T18:29:32.597 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:32.597 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:32.598 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:32.598 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:32.598 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:32.598 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:32.598 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:32.598 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:32.598 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:32.598 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:32.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.596+0000 7f236c3e8640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:32.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.596+0000 7f236c3e8640 4 set_mon_vals no callback set 2024-03-25T18:29:32.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.596+0000 7f236c3e8640 10 monclient: shutdown 2024-03-25T18:29:32.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.596+0000 7f236c3e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:32.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.596+0000 7f236c3e8640 1 -- 172.21.15.99:0/303944418 >> v1:172.21.15.190:6789/0 conn(0x7f23641430c0 legacy=0x7f236414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:32.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.597+0000 7f236c3e8640 1 -- 172.21.15.99:0/303944418 shutdown_connections 2024-03-25T18:29:32.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.597+0000 7f236c3e8640 1 -- 172.21.15.99:0/303944418 >> 172.21.15.99:0/303944418 conn(0x7f23640b7ea0 msgr2=0x7f23640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:32.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.597+0000 7f236c3e8640 1 -- 172.21.15.99:0/303944418 shutdown_connections 2024-03-25T18:29:32.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.597+0000 7f236c3e8640 1 -- 172.21.15.99:0/303944418 wait complete. 2024-03-25T18:29:32.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.598+0000 7f236c3e8640 10 monclient: build_initial_monmap 2024-03-25T18:29:32.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.598+0000 7f236c3e8640 10 monclient: monmap: 2024-03-25T18:29:32.602 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:32.602 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:32.602 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:29:32.602 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:29:32.602 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:32.602 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:29:32.602 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:32.602 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:29:32.603 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:29:32.603 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:32.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.598+0000 7f236c3e8640 1 librados: starting msgr at 2024-03-25T18:29:32.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.598+0000 7f236c3e8640 1 librados: starting objecter 2024-03-25T18:29:32.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.598+0000 7f236c3e8640 1 Processor -- start 2024-03-25T18:29:32.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.598+0000 7f236c3e8640 1 -- start start 2024-03-25T18:29:32.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.598+0000 7f236c3e8640 1 librados: setting wanted keys 2024-03-25T18:29:32.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.598+0000 7f236c3e8640 1 librados: calling monclient init 2024-03-25T18:29:32.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.598+0000 7f236c3e8640 10 monclient: init 2024-03-25T18:29:32.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.599+0000 7f236c3e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:32.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.599+0000 7f236c3e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:32.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.599+0000 7f236c3e8640 10 monclient(hunting): picked mon.noname-b con 0x7f23640a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:32.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.599+0000 7f236c3e8640 10 monclient(hunting): picked mon.noname-a con 0x7f236413f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:32.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.599+0000 7f236c3e8640 10 monclient(hunting): picked mon.noname-c con 0x7f23641430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:32.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.599+0000 7f236c3e8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23641d7c10 con 0x7f236413f710 2024-03-25T18:29:32.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.599+0000 7f236c3e8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2364137330 con 0x7f23640a98f0 2024-03-25T18:29:32.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.599+0000 7f236c3e8640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2364136890 con 0x7f23641430c0 2024-03-25T18:29:32.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.599+0000 7f236c3e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:32.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.599+0000 7f236c3e8640 10 monclient(hunting): authenticate will time out at 7590.767318s 2024-03-25T18:29:32.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.618+0000 7f2369983640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f236413f710 0x7f23641d7500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:49658/0 (socket says 172.21.15.99:49658) 2024-03-25T18:29:32.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.618+0000 7f2369983640 1 -- 172.21.15.99:0/2869507670 learned_addr learned my addr 172.21.15.99:0/2869507670 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:32.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.620+0000 7f235affd640 10 client.?.objecter ms_handle_connect 0x7f236413f710 2024-03-25T18:29:32.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.620+0000 7f235affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:32.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.622+0000 7f235affd640 10 client.?.objecter ms_handle_connect 0x7f23640a98f0 2024-03-25T18:29:32.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.622+0000 7f235affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:32.627 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.626+0000 7f235affd640 10 client.?.objecter ms_handle_connect 0x7f23641430c0 2024-03-25T18:29:32.627 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.626+0000 7f235affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:32.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.631+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628014006 0 0) 0x7f23641d7c10 con 0x7f236413f710 2024-03-25T18:29:32.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.631+0000 7f235affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:32.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.631+0000 7f235affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:32.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.631+0000 7f235affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:32.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.631+0000 7f235affd640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:29:32.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.631+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2348004eb0 con 0x7f236413f710 2024-03-25T18:29:32.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.634+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958429406 0 0) 0x7f2364137330 con 0x7f23640a98f0 2024-03-25T18:29:32.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.634+0000 7f235affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:32.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.634+0000 7f235affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:32.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.634+0000 7f235affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:32.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.634+0000 7f235affd640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:29:32.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.634+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23641d7c10 con 0x7f23640a98f0 2024-03-25T18:29:32.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.646+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3191902228 0 0) 0x7f2348004eb0 con 0x7f236413f710 2024-03-25T18:29:32.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.646+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2364137330 con 0x7f236413f710 2024-03-25T18:29:32.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.646+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f23540019d0 con 0x7f236413f710 2024-03-25T18:29:32.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.646+0000 7f235affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:32.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.646+0000 7f235affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:32.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.646+0000 7f235affd640 10 monclient(hunting): dump: 2024-03-25T18:29:32.648 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:32.648 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:32.648 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:32.648 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:32.648 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:32.648 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:32.648 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:32.648 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:32.648 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:32.648 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:32.649 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.646+0000 7f235affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:32.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.655+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047233100 0 0) 0x7f2364136890 con 0x7f23641430c0 2024-03-25T18:29:32.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.655+0000 7f235affd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:32.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.655+0000 7f235affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:32.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.655+0000 7f235affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:32.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.655+0000 7f235affd640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:29:32.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.655+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2348004eb0 con 0x7f23641430c0 2024-03-25T18:29:32.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.656+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3817415006 0 0) 0x7f23641d7c10 con 0x7f23640a98f0 2024-03-25T18:29:32.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.656+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2364136890 con 0x7f23640a98f0 2024-03-25T18:29:32.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.656+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2360001cb0 con 0x7f23640a98f0 2024-03-25T18:29:32.658 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.656+0000 7f235affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:32.658 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.656+0000 7f235affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:32.658 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.656+0000 7f235affd640 10 monclient(hunting): dump: 2024-03-25T18:29:32.658 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:32.658 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:32.658 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:32.658 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:32.658 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:32.658 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:32.658 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:32.658 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:32.658 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:32.658 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:32.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2662886229 0 0) 0x7f2364137330 con 0x7f236413f710 2024-03-25T18:29:32.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 >> v1:172.21.15.99:6790/0 conn(0x7f23641430c0 legacy=0x7f2364132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:32.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 >> v1:172.21.15.190:6789/0 conn(0x7f23640a98f0 legacy=0x7f23640b6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:32.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f235affd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:32.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f235affd640 1 monclient: found mon.a 2024-03-25T18:29:32.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f235affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:32.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f235affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:32.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2364136a20 con 0x7f236413f710 2024-03-25T18:29:32.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f235affd640 10 monclient: _finish_auth 0 2024-03-25T18:29:32.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f235affd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:32.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f235affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:32.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f236c3e8640 5 monclient: authenticate success, global_id 4398 2024-03-25T18:29:32.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f236c3e8640 10 monclient: _renew_subs 2024-03-25T18:29:32.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f236c3e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:32.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f236c3e8640 1 -- 172.21.15.99:0/2869507670 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2364139610 con 0x7f236413f710 2024-03-25T18:29:32.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f236c3e8640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:32.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f236c3e8640 10 monclient: _renew_subs 2024-03-25T18:29:32.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f236c3e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:32.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f236c3e8640 1 -- 172.21.15.99:0/2869507670 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2364137a30 con 0x7f236413f710 2024-03-25T18:29:32.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.667+0000 7f236c3e8640 1 librados: init done 2024-03-25T18:29:32.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.668+0000 7f236c3e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:32.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.668+0000 7f236c3e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:32.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.668+0000 7f236c3e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:32.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.668+0000 7f236c3e8640 1 -- 172.21.15.99:0/2869507670 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23641399b0 con 0x7f236413f710 2024-03-25T18:29:32.960 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.959+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2354003610 con 0x7f236413f710 2024-03-25T18:29:32.960 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.959+0000 7f235affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:32.960 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.959+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f23540039a0 con 0x7f236413f710 2024-03-25T18:29:32.961 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.959+0000 7f235affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:32.961 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.959+0000 7f235affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:32.961 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.959+0000 7f235affd640 10 monclient: dump: 2024-03-25T18:29:32.961 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:32.961 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:32.961 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:32.961 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:32.961 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:32.961 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:32.961 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:32.961 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:32.961 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:32.961 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:32.961 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.959+0000 7f2368981640 4 set_mon_vals no callback set 2024-03-25T18:29:32.962 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.960+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f2354003bc0 con 0x7f236413f710 2024-03-25T18:29:32.962 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.961+0000 7f236a184640 10 monclient: get_auth_request con 0x7f234806df60 auth_method 0 2024-03-25T18:29:32.962 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.961+0000 7f235affd640 10 client.4398.objecter ms_handle_connect 0x7f234806df60 2024-03-25T18:29:32.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.966+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f2354054740 con 0x7f236413f710 2024-03-25T18:29:32.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.966+0000 7f235affd640 10 client.4398.objecter ms_dispatch 0x7f2364000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:32.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.966+0000 7f235affd640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:32.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.966+0000 7f235affd640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:32.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.966+0000 7f235affd640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-25T18:29:32.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.966+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2354050250 con 0x7f236413f710 2024-03-25T18:29:32.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.966+0000 7f235affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:32.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:32.966+0000 7f235affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:33.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.167+0000 7f236c3e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:33.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.167+0000 7f236c3e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:33.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.167+0000 7f236c3e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:33.169 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.167+0000 7f236c3e8640 1 -- 172.21.15.99:0/2869507670 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23640627b0 con 0x7f236413f710 2024-03-25T18:29:33.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.217+0000 7f235affd640 1 -- 172.21.15.99:0/2869507670 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f2354052180 con 0x7f236413f710 2024-03-25T18:29:33.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.217+0000 7f235affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:33.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.217+0000 7f235affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:33.219 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:29:33.219 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:33.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.219+0000 7f236c3e8640 10 librados: watch_flush enter 2024-03-25T18:29:33.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.219+0000 7f236c3e8640 10 librados: watch_flush exit 2024-03-25T18:29:33.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.219+0000 7f236c3e8640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-25T18:29:33.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.219+0000 7f236c3e8640 10 client.4398.objecter successfully canceled tick 2024-03-25T18:29:33.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.220+0000 7f236c3e8640 1 -- 172.21.15.99:0/2869507670 >> v1:172.21.15.190:6800/4220409699 conn(0x7f234806df60 legacy=0x7f2348070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:33.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.220+0000 7f236c3e8640 10 monclient: shutdown 2024-03-25T18:29:33.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.220+0000 7f236c3e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:33.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.220+0000 7f236c3e8640 1 -- 172.21.15.99:0/2869507670 >> v1:172.21.15.99:6789/0 conn(0x7f236413f710 legacy=0x7f23641d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:33.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.220+0000 7f236c3e8640 1 -- 172.21.15.99:0/2869507670 shutdown_connections 2024-03-25T18:29:33.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.220+0000 7f236c3e8640 1 -- 172.21.15.99:0/2869507670 >> 172.21.15.99:0/2869507670 conn(0x7f23640b7ea0 msgr2=0x7f2364074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:33.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.220+0000 7f236c3e8640 1 -- 172.21.15.99:0/2869507670 shutdown_connections 2024-03-25T18:29:33.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.220+0000 7f236c3e8640 1 -- 172.21.15.99:0/2869507670 wait complete. 2024-03-25T18:29:33.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:33.220+0000 7f236c3e8640 1 librados: shutdown 2024-03-25T18:29:33.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:40.040 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:40.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.158+0000 7f48ffc06640 10 monclient: get_monmap_and_config 2024-03-25T18:29:40.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.158+0000 7f48ffc06640 10 monclient: build_initial_monmap 2024-03-25T18:29:40.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.158+0000 7f48ffc06640 10 monclient: monmap: 2024-03-25T18:29:40.160 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:40.160 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:40.161 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:29:40.159123+0000 2024-03-25T18:29:40.161 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:29:40.159123+0000 2024-03-25T18:29:40.161 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:40.161 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:40.161 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:40.161 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:29:40.161 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:29:40.162 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:40.162 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.158+0000 7f48ffc06640 1 Processor -- start 2024-03-25T18:29:40.162 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.158+0000 7f48ffc06640 1 -- start start 2024-03-25T18:29:40.162 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.158+0000 7f48ffc06640 10 monclient: init 2024-03-25T18:29:40.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.159+0000 7f48ffc06640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:40.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.159+0000 7f48ffc06640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:40.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.159+0000 7f48ffc06640 10 monclient(hunting): picked mon.noname-a con 0x7f48f80a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:40.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.159+0000 7f48ffc06640 10 monclient(hunting): picked mon.noname-c con 0x7f48f813f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:40.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.159+0000 7f48ffc06640 10 monclient(hunting): picked mon.noname-b con 0x7f48f81430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:40.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.159+0000 7f48ffc06640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f8142520 con 0x7f48f80a98f0 2024-03-25T18:29:40.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.159+0000 7f48ffc06640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f80b28a0 con 0x7f48f813f710 2024-03-25T18:29:40.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.159+0000 7f48ffc06640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f80b2a30 con 0x7f48f81430c0 2024-03-25T18:29:40.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.159+0000 7f48ffc06640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:40.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.159+0000 7f48ffc06640 10 monclient(hunting): authenticate will time out at 7598.327657s 2024-03-25T18:29:40.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.164+0000 7f48fd1a1640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f48f813f710 0x7f48f81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:34778/0 (socket says 172.21.15.99:34778) 2024-03-25T18:29:40.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.164+0000 7f48fd1a1640 1 -- 172.21.15.99:0/2799838825 learned_addr learned my addr 172.21.15.99:0/2799838825 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:40.187 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.186+0000 7f48fc9a0640 1 -- 172.21.15.99:0/2799838825 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939705764 0 0) 0x7f48f80b2a30 con 0x7f48f81430c0 2024-03-25T18:29:40.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.186+0000 7f48fc9a0640 10 monclient(hunting): _negotiate 2024-03-25T18:29:40.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.186+0000 7f48fc9a0640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:40.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.186+0000 7f48fc9a0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:40.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.186+0000 7f48fc9a0640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:29:40.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.186+0000 7f48fc9a0640 1 -- 172.21.15.99:0/2799838825 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48e4003520 con 0x7f48f81430c0 2024-03-25T18:29:40.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.187+0000 7f48fc9a0640 1 -- 172.21.15.99:0/2799838825 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957321288 0 0) 0x7f48f80b28a0 con 0x7f48f813f710 2024-03-25T18:29:40.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.187+0000 7f48fc9a0640 10 monclient(hunting): _negotiate 2024-03-25T18:29:40.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.187+0000 7f48fc9a0640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:40.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.187+0000 7f48fc9a0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:40.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.187+0000 7f48fc9a0640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:29:40.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.187+0000 7f48fc9a0640 1 -- 172.21.15.99:0/2799838825 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f80b2a30 con 0x7f48f813f710 2024-03-25T18:29:40.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48fc9a0640 1 -- 172.21.15.99:0/2799838825 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1002240093 0 0) 0x7f48f80b2a30 con 0x7f48f813f710 2024-03-25T18:29:40.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48fc9a0640 1 -- 172.21.15.99:0/2799838825 >> v1:172.21.15.99:6790/0 conn(0x7f48f81430c0 legacy=0x7f48f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:40.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48fc9a0640 1 -- 172.21.15.99:0/2799838825 >> v1:172.21.15.99:6789/0 conn(0x7f48f80a98f0 legacy=0x7f48f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:40.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48fc9a0640 10 monclient: _finish_hunting 0 2024-03-25T18:29:40.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48fc9a0640 1 monclient: found mon.noname-c 2024-03-25T18:29:40.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48fc9a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:40.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48fc9a0640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:29:40.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48fc9a0640 1 -- 172.21.15.99:0/2799838825 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f80b44c0 con 0x7f48f813f710 2024-03-25T18:29:40.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48fc9a0640 10 monclient: _finish_auth 0 2024-03-25T18:29:40.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48fc9a0640 10 monclient: _check_auth_tickets 2024-03-25T18:29:40.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48fc9a0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:40.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48fc9a0640 1 -- 172.21.15.99:0/2799838825 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f48e8001670 con 0x7f48f813f710 2024-03-25T18:29:40.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48ffc06640 5 monclient: authenticate success, global_id 4417 2024-03-25T18:29:40.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48ffc06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:40.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.196+0000 7f48fc9a0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:40.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.197+0000 7f48fc9a0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:40.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.197+0000 7f48fc9a0640 10 monclient: dump: 2024-03-25T18:29:40.200 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:40.200 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:40.200 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:40.200 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:40.200 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:40.200 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:40.200 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:40.200 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:40.200 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:40.200 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:40.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.197+0000 7f48ffc06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:40.202 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.202+0000 7f48fc9a0640 1 -- 172.21.15.99:0/2799838825 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48e80017d0 con 0x7f48f813f710 2024-03-25T18:29:40.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.202+0000 7f48fc9a0640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:40.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.202+0000 7f48ffc06640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:40.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.202+0000 7f48ffc06640 4 set_mon_vals no callback set 2024-03-25T18:29:40.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.202+0000 7f48fc9a0640 1 -- 172.21.15.99:0/2799838825 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f48e8001ba0 con 0x7f48f813f710 2024-03-25T18:29:40.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.202+0000 7f48fc9a0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:40.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.202+0000 7f48fc9a0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:40.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.202+0000 7f48fc9a0640 10 monclient: dump: 2024-03-25T18:29:40.204 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:40.204 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:40.204 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:40.204 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:40.204 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:40.204 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:40.205 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:40.205 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:40.205 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:40.205 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:40.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.202+0000 7f48ffc06640 10 monclient: shutdown 2024-03-25T18:29:40.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.202+0000 7f48ffc06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:40.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.202+0000 7f48ffc06640 1 -- 172.21.15.99:0/2799838825 >> v1:172.21.15.190:6789/0 conn(0x7f48f813f710 legacy=0x7f48f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:40.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.203+0000 7f48ffc06640 1 -- 172.21.15.99:0/2799838825 shutdown_connections 2024-03-25T18:29:40.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.203+0000 7f48ffc06640 1 -- 172.21.15.99:0/2799838825 >> 172.21.15.99:0/2799838825 conn(0x7f48f80b7ea0 msgr2=0x7f48f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:40.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.203+0000 7f48ffc06640 1 -- 172.21.15.99:0/2799838825 shutdown_connections 2024-03-25T18:29:40.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.203+0000 7f48ffc06640 1 -- 172.21.15.99:0/2799838825 wait complete. 2024-03-25T18:29:40.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.204+0000 7f48ffc06640 10 monclient: build_initial_monmap 2024-03-25T18:29:40.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.204+0000 7f48ffc06640 10 monclient: monmap: 2024-03-25T18:29:40.207 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:40.207 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:40.207 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:29:40.207 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:29:40.207 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:40.207 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:29:40.207 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:40.207 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:29:40.207 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:29:40.207 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:40.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.204+0000 7f48ffc06640 1 librados: starting msgr at 2024-03-25T18:29:40.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.204+0000 7f48ffc06640 1 librados: starting objecter 2024-03-25T18:29:40.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.204+0000 7f48ffc06640 1 Processor -- start 2024-03-25T18:29:40.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.204+0000 7f48ffc06640 1 -- start start 2024-03-25T18:29:40.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.204+0000 7f48ffc06640 1 librados: setting wanted keys 2024-03-25T18:29:40.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.204+0000 7f48ffc06640 1 librados: calling monclient init 2024-03-25T18:29:40.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.204+0000 7f48ffc06640 10 monclient: init 2024-03-25T18:29:40.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.205+0000 7f48ffc06640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:40.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.205+0000 7f48ffc06640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:40.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.205+0000 7f48ffc06640 10 monclient(hunting): picked mon.noname-c con 0x7f48f80a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:40.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.205+0000 7f48ffc06640 10 monclient(hunting): picked mon.noname-b con 0x7f48f813f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:40.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.205+0000 7f48ffc06640 10 monclient(hunting): picked mon.noname-a con 0x7f48f81430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:40.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.205+0000 7f48ffc06640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f81399a0 con 0x7f48f81430c0 2024-03-25T18:29:40.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.205+0000 7f48ffc06640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f80ab8b0 con 0x7f48f813f710 2024-03-25T18:29:40.209 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.205+0000 7f48ffc06640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f81388e0 con 0x7f48f80a98f0 2024-03-25T18:29:40.209 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.205+0000 7f48ffc06640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:40.209 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.205+0000 7f48ffc06640 10 monclient(hunting): authenticate will time out at 7598.373476s 2024-03-25T18:29:40.221 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.220+0000 7f48fe1a3640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f48f81430c0 0x7f48f81355f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:48088/0 (socket says 172.21.15.99:48088) 2024-03-25T18:29:40.221 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.220+0000 7f48fe1a3640 1 -- 172.21.15.99:0/363034264 learned_addr learned my addr 172.21.15.99:0/363034264 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:40.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.222+0000 7f48ee7fc640 10 client.?.objecter ms_handle_connect 0x7f48f813f710 2024-03-25T18:29:40.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.222+0000 7f48ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:40.226 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.225+0000 7f48ee7fc640 10 client.?.objecter ms_handle_connect 0x7f48f81430c0 2024-03-25T18:29:40.226 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.225+0000 7f48ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:40.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.231+0000 7f48ee7fc640 10 client.?.objecter ms_handle_connect 0x7f48f80a98f0 2024-03-25T18:29:40.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.231+0000 7f48ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:40.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.231+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405475718 0 0) 0x7f48f80ab8b0 con 0x7f48f813f710 2024-03-25T18:29:40.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.231+0000 7f48ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:40.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.231+0000 7f48ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:40.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.231+0000 7f48ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:40.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.231+0000 7f48ee7fc640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:29:40.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.231+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48dc004eb0 con 0x7f48f813f710 2024-03-25T18:29:40.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.241+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1335684304 0 0) 0x7f48dc004eb0 con 0x7f48f813f710 2024-03-25T18:29:40.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.241+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48f80ab8b0 con 0x7f48f813f710 2024-03-25T18:29:40.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.241+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f48e8001530 con 0x7f48f813f710 2024-03-25T18:29:40.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.241+0000 7f48ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:40.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.241+0000 7f48ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:40.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.241+0000 7f48ee7fc640 10 monclient(hunting): dump: 2024-03-25T18:29:40.243 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:40.243 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:40.243 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:40.243 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:40.243 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:40.243 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:40.243 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:40.244 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:40.244 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:40.244 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:40.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.241+0000 7f48ee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:40.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.246+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634709915 0 0) 0x7f48f81399a0 con 0x7f48f81430c0 2024-03-25T18:29:40.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.246+0000 7f48ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:40.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.246+0000 7f48ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:40.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.246+0000 7f48ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:40.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.246+0000 7f48ee7fc640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:29:40.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.246+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48dc004eb0 con 0x7f48f81430c0 2024-03-25T18:29:40.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.248+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168001515 0 0) 0x7f48f81388e0 con 0x7f48f80a98f0 2024-03-25T18:29:40.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.248+0000 7f48ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:40.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.248+0000 7f48ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:40.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.248+0000 7f48ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:40.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.248+0000 7f48ee7fc640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:29:40.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.248+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f81399a0 con 0x7f48f80a98f0 2024-03-25T18:29:40.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3811834299 0 0) 0x7f48dc004eb0 con 0x7f48f81430c0 2024-03-25T18:29:40.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48f81388e0 con 0x7f48f81430c0 2024-03-25T18:29:40.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f48f0001c30 con 0x7f48f81430c0 2024-03-25T18:29:40.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:40.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:40.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 10 monclient(hunting): dump: 2024-03-25T18:29:40.258 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:40.258 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:40.259 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:40.259 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:40.259 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:40.259 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:40.259 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:40.259 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:40.259 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:40.259 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:40.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 814485086 0 0) 0x7f48f80ab8b0 con 0x7f48f813f710 2024-03-25T18:29:40.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 >> v1:172.21.15.99:6790/0 conn(0x7f48f80a98f0 legacy=0x7f48f8133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:40.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 >> v1:172.21.15.99:6789/0 conn(0x7f48f81430c0 legacy=0x7f48f81355f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:40.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:40.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 1 monclient: found mon.b 2024-03-25T18:29:40.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:40.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:40.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f80b13f0 con 0x7f48f813f710 2024-03-25T18:29:40.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:40.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:40.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:40.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ffc06640 5 monclient: authenticate success, global_id 4420 2024-03-25T18:29:40.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.257+0000 7f48ffc06640 10 monclient: _renew_subs 2024-03-25T18:29:40.262 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.258+0000 7f48ffc06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:40.262 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.258+0000 7f48ffc06640 1 -- 172.21.15.99:0/363034264 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f81385b0 con 0x7f48f813f710 2024-03-25T18:29:40.262 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.258+0000 7f48ffc06640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:40.262 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.258+0000 7f48ffc06640 10 monclient: _renew_subs 2024-03-25T18:29:40.262 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.258+0000 7f48ffc06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:40.262 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.258+0000 7f48ffc06640 1 -- 172.21.15.99:0/363034264 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f81377b0 con 0x7f48f813f710 2024-03-25T18:29:40.262 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.258+0000 7f48ffc06640 1 librados: init done 2024-03-25T18:29:40.262 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.259+0000 7f48ffc06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:40.262 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.259+0000 7f48ffc06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:40.262 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.259+0000 7f48ffc06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:40.262 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.259+0000 7f48ffc06640 1 -- 172.21.15.99:0/363034264 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f80627b0 con 0x7f48f813f710 2024-03-25T18:29:40.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.535+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48e8004a90 con 0x7f48f813f710 2024-03-25T18:29:40.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.535+0000 7f48ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:40.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.535+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f48e8004da0 con 0x7f48f813f710 2024-03-25T18:29:40.537 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.535+0000 7f48ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:40.537 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.535+0000 7f48ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:40.537 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.535+0000 7f48ee7fc640 10 monclient: dump: 2024-03-25T18:29:40.537 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:40.537 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:40.537 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:40.537 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:40.537 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:40.537 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:40.537 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:40.537 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:40.537 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:40.537 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:40.538 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.535+0000 7f48effff640 4 set_mon_vals no callback set 2024-03-25T18:29:40.538 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.536+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f48e801ad70 con 0x7f48f813f710 2024-03-25T18:29:40.538 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.536+0000 7f48fd9a2640 10 monclient: get_auth_request con 0x7f48dc06df40 auth_method 0 2024-03-25T18:29:40.539 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.536+0000 7f48ee7fc640 10 client.4420.objecter ms_handle_connect 0x7f48dc06df40 2024-03-25T18:29:40.541 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.540+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f48e8054940 con 0x7f48f813f710 2024-03-25T18:29:40.541 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.540+0000 7f48ee7fc640 10 client.4420.objecter ms_dispatch 0x7f48f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:40.542 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.540+0000 7f48ee7fc640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:40.542 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.540+0000 7f48ee7fc640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:40.542 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.540+0000 7f48ee7fc640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-25T18:29:40.542 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.540+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f48e8050280 con 0x7f48f813f710 2024-03-25T18:29:40.542 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.540+0000 7f48ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:40.542 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.540+0000 7f48ee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:40.805 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.803+0000 7f48ffc06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:40.805 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.803+0000 7f48ffc06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:40.805 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.803+0000 7f48ffc06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:29:40.805 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.803+0000 7f48ffc06640 1 -- 172.21.15.99:0/363034264 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48f8074cc0 con 0x7f48f813f710 2024-03-25T18:29:40.831 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.830+0000 7f48ee7fc640 1 -- 172.21.15.99:0/363034264 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f48e8052380 con 0x7f48f813f710 2024-03-25T18:29:40.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.830+0000 7f48ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:40.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.830+0000 7f48ee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:40.832 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:29:40.832 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:40.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.833+0000 7f48ffc06640 10 librados: watch_flush enter 2024-03-25T18:29:40.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.833+0000 7f48ffc06640 10 librados: watch_flush exit 2024-03-25T18:29:40.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.833+0000 7f48ffc06640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-25T18:29:40.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.833+0000 7f48ffc06640 10 client.4420.objecter successfully canceled tick 2024-03-25T18:29:40.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.833+0000 7f48ffc06640 1 -- 172.21.15.99:0/363034264 >> v1:172.21.15.190:6800/4220409699 conn(0x7f48dc06df40 legacy=0x7f48dc070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:40.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.833+0000 7f48ffc06640 10 monclient: shutdown 2024-03-25T18:29:40.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.833+0000 7f48ffc06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:40.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.833+0000 7f48ffc06640 1 -- 172.21.15.99:0/363034264 >> v1:172.21.15.190:6789/0 conn(0x7f48f813f710 legacy=0x7f48f8134ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:40.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.833+0000 7f48ffc06640 1 -- 172.21.15.99:0/363034264 shutdown_connections 2024-03-25T18:29:40.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.833+0000 7f48ffc06640 1 -- 172.21.15.99:0/363034264 >> 172.21.15.99:0/363034264 conn(0x7f48f80b7ea0 msgr2=0x7f48f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:40.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.833+0000 7f48ffc06640 1 -- 172.21.15.99:0/363034264 shutdown_connections 2024-03-25T18:29:40.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.833+0000 7f48ffc06640 1 -- 172.21.15.99:0/363034264 wait complete. 2024-03-25T18:29:40.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:40.833+0000 7f48ffc06640 1 librados: shutdown 2024-03-25T18:29:40.847 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:47.655 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:47.773 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.771+0000 7fda42063640 10 monclient: get_monmap_and_config 2024-03-25T18:29:47.773 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.771+0000 7fda42063640 10 monclient: build_initial_monmap 2024-03-25T18:29:47.773 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.771+0000 7fda42063640 10 monclient: monmap: 2024-03-25T18:29:47.773 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:47.774 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:47.774 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:29:47.772538+0000 2024-03-25T18:29:47.774 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:29:47.772538+0000 2024-03-25T18:29:47.774 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:47.774 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:47.774 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:47.774 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:29:47.774 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:29:47.774 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:47.774 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.772+0000 7fda42063640 1 Processor -- start 2024-03-25T18:29:47.774 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.772+0000 7fda42063640 1 -- start start 2024-03-25T18:29:47.774 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.772+0000 7fda42063640 10 monclient: init 2024-03-25T18:29:47.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.773+0000 7fda42063640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:47.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.773+0000 7fda42063640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:47.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.773+0000 7fda42063640 10 monclient(hunting): picked mon.noname-a con 0x7fda3c0a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:47.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.773+0000 7fda42063640 10 monclient(hunting): picked mon.noname-c con 0x7fda3c13f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:47.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.773+0000 7fda42063640 10 monclient(hunting): picked mon.noname-b con 0x7fda3c1430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:47.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.773+0000 7fda42063640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c142520 con 0x7fda3c0a98f0 2024-03-25T18:29:47.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.773+0000 7fda42063640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c0b28a0 con 0x7fda3c13f710 2024-03-25T18:29:47.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.773+0000 7fda42063640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c0b2a30 con 0x7fda3c1430c0 2024-03-25T18:29:47.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.773+0000 7fda42063640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:47.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.773+0000 7fda42063640 10 monclient(hunting): authenticate will time out at 7605.941231s 2024-03-25T18:29:47.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.775+0000 7fda3bfff640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7fda3c1430c0 0x7fda3c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:56308/0 (socket says 172.21.15.99:56308) 2024-03-25T18:29:47.777 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.775+0000 7fda3bfff640 1 -- 172.21.15.99:0/397448140 learned_addr learned my addr 172.21.15.99:0/397448140 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:47.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.792+0000 7fda3a7fc640 1 -- 172.21.15.99:0/397448140 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40379239 0 0) 0x7fda3c0b2a30 con 0x7fda3c1430c0 2024-03-25T18:29:47.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.792+0000 7fda3a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:47.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.792+0000 7fda3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:47.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.792+0000 7fda3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:47.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.792+0000 7fda3a7fc640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:29:47.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.792+0000 7fda3a7fc640 1 -- 172.21.15.99:0/397448140 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda24003520 con 0x7fda3c1430c0 2024-03-25T18:29:47.810 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.806+0000 7fda3a7fc640 1 -- 172.21.15.99:0/397448140 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312277712 0 0) 0x7fda3c142520 con 0x7fda3c0a98f0 2024-03-25T18:29:47.810 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.806+0000 7fda3a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:47.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.806+0000 7fda3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:47.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.806+0000 7fda3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:47.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.806+0000 7fda3a7fc640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:29:47.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.806+0000 7fda3a7fc640 1 -- 172.21.15.99:0/397448140 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda3c0b2a30 con 0x7fda3c0a98f0 2024-03-25T18:29:47.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 1 -- 172.21.15.99:0/397448140 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3971547136 0 0) 0x7fda3c0b2a30 con 0x7fda3c0a98f0 2024-03-25T18:29:47.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 1 -- 172.21.15.99:0/397448140 >> v1:172.21.15.99:6790/0 conn(0x7fda3c1430c0 legacy=0x7fda3c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:47.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 1 -- 172.21.15.99:0/397448140 >> v1:172.21.15.190:6789/0 conn(0x7fda3c13f710 legacy=0x7fda3c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:47.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:47.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:29:47.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:47.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:29:47.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 1 -- 172.21.15.99:0/397448140 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c0b44c0 con 0x7fda3c0a98f0 2024-03-25T18:29:47.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:47.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:47.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:47.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 1 -- 172.21.15.99:0/397448140 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fda2c001670 con 0x7fda3c0a98f0 2024-03-25T18:29:47.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda42063640 5 monclient: authenticate success, global_id 4407 2024-03-25T18:29:47.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda42063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:47.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:47.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:47.821 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda3a7fc640 10 monclient: dump: 2024-03-25T18:29:47.821 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:47.821 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:47.822 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:47.822 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:47.822 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:47.822 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:47.822 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:47.822 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:47.822 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:47.822 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:47.822 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.819+0000 7fda42063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:47.881 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.880+0000 7fda3a7fc640 1 -- 172.21.15.99:0/397448140 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda2c0017d0 con 0x7fda3c0a98f0 2024-03-25T18:29:47.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.880+0000 7fda3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:47.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.880+0000 7fda3a7fc640 1 -- 172.21.15.99:0/397448140 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fda2c001ba0 con 0x7fda3c0a98f0 2024-03-25T18:29:47.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.880+0000 7fda42063640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:47.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.880+0000 7fda42063640 4 set_mon_vals no callback set 2024-03-25T18:29:47.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.880+0000 7fda3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:47.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.880+0000 7fda3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:47.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.880+0000 7fda3a7fc640 10 monclient: dump: 2024-03-25T18:29:47.882 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:47.882 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:47.882 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:47.882 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:47.882 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:47.883 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:47.883 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:47.883 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:47.883 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:47.883 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:47.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.881+0000 7fda42063640 10 monclient: shutdown 2024-03-25T18:29:47.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.881+0000 7fda42063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:47.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.881+0000 7fda42063640 1 -- 172.21.15.99:0/397448140 >> v1:172.21.15.99:6789/0 conn(0x7fda3c0a98f0 legacy=0x7fda3c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:47.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.881+0000 7fda42063640 1 -- 172.21.15.99:0/397448140 shutdown_connections 2024-03-25T18:29:47.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.881+0000 7fda42063640 1 -- 172.21.15.99:0/397448140 >> 172.21.15.99:0/397448140 conn(0x7fda3c0b7ea0 msgr2=0x7fda3c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:47.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.881+0000 7fda42063640 1 -- 172.21.15.99:0/397448140 shutdown_connections 2024-03-25T18:29:47.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.881+0000 7fda42063640 1 -- 172.21.15.99:0/397448140 wait complete. 2024-03-25T18:29:47.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.882+0000 7fda42063640 10 monclient: build_initial_monmap 2024-03-25T18:29:47.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.882+0000 7fda42063640 10 monclient: monmap: 2024-03-25T18:29:47.885 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:47.885 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:47.885 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:29:47.886 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:29:47.886 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:47.886 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:29:47.886 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:47.886 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:29:47.886 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:29:47.886 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:47.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.882+0000 7fda42063640 1 librados: starting msgr at 2024-03-25T18:29:47.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.882+0000 7fda42063640 1 librados: starting objecter 2024-03-25T18:29:47.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.882+0000 7fda42063640 1 Processor -- start 2024-03-25T18:29:47.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.882+0000 7fda42063640 1 -- start start 2024-03-25T18:29:47.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.882+0000 7fda42063640 1 librados: setting wanted keys 2024-03-25T18:29:47.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.882+0000 7fda42063640 1 librados: calling monclient init 2024-03-25T18:29:47.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.882+0000 7fda42063640 10 monclient: init 2024-03-25T18:29:47.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.883+0000 7fda42063640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:47.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.883+0000 7fda42063640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:47.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.883+0000 7fda42063640 10 monclient(hunting): picked mon.noname-b con 0x7fda3c0a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:47.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.883+0000 7fda42063640 10 monclient(hunting): picked mon.noname-a con 0x7fda3c13f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:47.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.883+0000 7fda42063640 10 monclient(hunting): picked mon.noname-c con 0x7fda3c1430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:47.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.883+0000 7fda42063640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c138630 con 0x7fda3c13f710 2024-03-25T18:29:47.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.883+0000 7fda42063640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c139280 con 0x7fda3c0a98f0 2024-03-25T18:29:47.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.883+0000 7fda42063640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c1349e0 con 0x7fda3c1430c0 2024-03-25T18:29:47.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.883+0000 7fda42063640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:47.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.883+0000 7fda42063640 10 monclient(hunting): authenticate will time out at 7606.051440s 2024-03-25T18:29:47.894 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.893+0000 7fda3affd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fda3c13f710 0x7fda3c133bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53730/0 (socket says 172.21.15.99:53730) 2024-03-25T18:29:47.894 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.893+0000 7fda3affd640 1 -- 172.21.15.99:0/2415728743 learned_addr learned my addr 172.21.15.99:0/2415728743 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:47.895 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.893+0000 7fda23fff640 10 client.?.objecter ms_handle_connect 0x7fda3c13f710 2024-03-25T18:29:47.895 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.894+0000 7fda23fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:47.917 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.916+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646812340 0 0) 0x7fda3c138630 con 0x7fda3c13f710 2024-03-25T18:29:47.918 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.916+0000 7fda23fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:47.918 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.916+0000 7fda23fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:47.918 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.916+0000 7fda23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:47.918 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.916+0000 7fda23fff640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:29:47.918 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.916+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda28004eb0 con 0x7fda3c13f710 2024-03-25T18:29:47.929 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.928+0000 7fda23fff640 10 client.?.objecter ms_handle_connect 0x7fda3c0a98f0 2024-03-25T18:29:47.929 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.928+0000 7fda23fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:47.939 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.938+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811800385 0 0) 0x7fda3c139280 con 0x7fda3c0a98f0 2024-03-25T18:29:47.939 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.938+0000 7fda23fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:47.939 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.938+0000 7fda23fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:47.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.938+0000 7fda23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:47.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.938+0000 7fda23fff640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:29:47.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.938+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda3c138630 con 0x7fda3c0a98f0 2024-03-25T18:29:47.942 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.941+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1744521810 0 0) 0x7fda28004eb0 con 0x7fda3c13f710 2024-03-25T18:29:47.942 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.941+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda3c139280 con 0x7fda3c13f710 2024-03-25T18:29:47.942 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.941+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fda30001a30 con 0x7fda3c13f710 2024-03-25T18:29:47.942 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.941+0000 7fda23fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:47.942 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.941+0000 7fda23fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:47.942 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.941+0000 7fda23fff640 10 monclient(hunting): dump: 2024-03-25T18:29:47.942 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:47.942 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:47.943 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:47.943 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:47.943 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:47.943 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:47.943 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:47.943 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:47.943 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:47.943 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:47.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.941+0000 7fda23fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:47.946 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.945+0000 7fda23fff640 10 client.?.objecter ms_handle_connect 0x7fda3c1430c0 2024-03-25T18:29:47.946 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.945+0000 7fda23fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:47.954 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.952+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 132474623 0 0) 0x7fda3c139280 con 0x7fda3c13f710 2024-03-25T18:29:47.954 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.952+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 >> v1:172.21.15.99:6790/0 conn(0x7fda3c1430c0 legacy=0x7fda3c1342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:47.954 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.952+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 >> v1:172.21.15.190:6789/0 conn(0x7fda3c0a98f0 legacy=0x7fda3c1d7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:47.954 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.952+0000 7fda23fff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:47.954 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.952+0000 7fda23fff640 1 monclient: found mon.a 2024-03-25T18:29:47.954 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.952+0000 7fda23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:47.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.953+0000 7fda23fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:47.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.953+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c139560 con 0x7fda3c13f710 2024-03-25T18:29:47.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.953+0000 7fda23fff640 10 monclient: _finish_auth 0 2024-03-25T18:29:47.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.953+0000 7fda23fff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:47.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.953+0000 7fda23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:47.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.953+0000 7fda42063640 5 monclient: authenticate success, global_id 4410 2024-03-25T18:29:47.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.953+0000 7fda42063640 10 monclient: _renew_subs 2024-03-25T18:29:47.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.953+0000 7fda42063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:47.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.953+0000 7fda42063640 1 -- 172.21.15.99:0/2415728743 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3c139840 con 0x7fda3c13f710 2024-03-25T18:29:47.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.953+0000 7fda42063640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:47.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.953+0000 7fda42063640 10 monclient: _renew_subs 2024-03-25T18:29:47.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.953+0000 7fda42063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:47.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.953+0000 7fda42063640 1 -- 172.21.15.99:0/2415728743 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c1381d0 con 0x7fda3c13f710 2024-03-25T18:29:47.955 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.953+0000 7fda42063640 1 librados: init done 2024-03-25T18:29:47.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.954+0000 7fda42063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:47.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.954+0000 7fda42063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:47.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.954+0000 7fda42063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:47.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:47.954+0000 7fda42063640 1 -- 172.21.15.99:0/2415728743 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c0627b0 con 0x7fda3c13f710 2024-03-25T18:29:48.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.252+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda30001c70 con 0x7fda3c13f710 2024-03-25T18:29:48.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.252+0000 7fda23fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:48.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.252+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fda30003a60 con 0x7fda3c13f710 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.252+0000 7fda23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.252+0000 7fda23fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.252+0000 7fda23fff640 10 monclient: dump: 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:48.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.252+0000 7fda39ffb640 4 set_mon_vals no callback set 2024-03-25T18:29:48.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.253+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fda30003c80 con 0x7fda3c13f710 2024-03-25T18:29:48.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.253+0000 7fda3b7fe640 10 monclient: get_auth_request con 0x7fda2806dd40 auth_method 0 2024-03-25T18:29:48.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.254+0000 7fda23fff640 10 client.4410.objecter ms_handle_connect 0x7fda2806dd40 2024-03-25T18:29:48.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.259+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fda30054820 con 0x7fda3c13f710 2024-03-25T18:29:48.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.259+0000 7fda23fff640 10 client.4410.objecter ms_dispatch 0x7fda3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:48.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.259+0000 7fda23fff640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:48.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.259+0000 7fda23fff640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:48.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.259+0000 7fda23fff640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-25T18:29:48.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.259+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda30050330 con 0x7fda3c13f710 2024-03-25T18:29:48.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.259+0000 7fda23fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:48.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.259+0000 7fda23fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:48.511 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.510+0000 7fda42063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:48.511 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.510+0000 7fda42063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:48.511 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.510+0000 7fda42063640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:29:48.511 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.510+0000 7fda42063640 1 -- 172.21.15.99:0/2415728743 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda3c1387e0 con 0x7fda3c13f710 2024-03-25T18:29:48.548 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.546+0000 7fda23fff640 1 -- 172.21.15.99:0/2415728743 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fda3c1387e0 con 0x7fda3c13f710 2024-03-25T18:29:48.548 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.546+0000 7fda23fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:48.548 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.546+0000 7fda23fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:48.548 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:29:48.548 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:48.551 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.548+0000 7fda42063640 10 librados: watch_flush enter 2024-03-25T18:29:48.551 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.548+0000 7fda42063640 10 librados: watch_flush exit 2024-03-25T18:29:48.551 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.548+0000 7fda42063640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-25T18:29:48.551 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.548+0000 7fda42063640 10 client.4410.objecter successfully canceled tick 2024-03-25T18:29:48.551 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.549+0000 7fda42063640 1 -- 172.21.15.99:0/2415728743 >> v1:172.21.15.190:6800/4220409699 conn(0x7fda2806dd40 legacy=0x7fda28070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:48.551 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.549+0000 7fda42063640 10 monclient: shutdown 2024-03-25T18:29:48.551 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.549+0000 7fda42063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:48.551 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.549+0000 7fda42063640 1 -- 172.21.15.99:0/2415728743 >> v1:172.21.15.99:6789/0 conn(0x7fda3c13f710 legacy=0x7fda3c133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:48.552 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.549+0000 7fda42063640 1 -- 172.21.15.99:0/2415728743 shutdown_connections 2024-03-25T18:29:48.552 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.549+0000 7fda42063640 1 -- 172.21.15.99:0/2415728743 >> 172.21.15.99:0/2415728743 conn(0x7fda3c0b7ea0 msgr2=0x7fda3c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:48.552 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.549+0000 7fda42063640 1 -- 172.21.15.99:0/2415728743 shutdown_connections 2024-03-25T18:29:48.552 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.549+0000 7fda42063640 1 -- 172.21.15.99:0/2415728743 wait complete. 2024-03-25T18:29:48.552 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:48.549+0000 7fda42063640 1 librados: shutdown 2024-03-25T18:29:48.562 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:49.077 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:04:49.669 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:49.136 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:04:49.728 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:49.168 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:04:49.760 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:49.178 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:04:49.737 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:49.179 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:04:49.772 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:49.181 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:04:49.740 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:49.187 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:04:49.779 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:49.215 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:04:49.774 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:49.229 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:04:49.822 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:55.369 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:55.479 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.478+0000 7f7b6357b640 10 monclient: get_monmap_and_config 2024-03-25T18:29:55.479 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.478+0000 7f7b6357b640 10 monclient: build_initial_monmap 2024-03-25T18:29:55.479 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.478+0000 7f7b6357b640 10 monclient: monmap: 2024-03-25T18:29:55.479 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:55.479 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:55.480 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:29:55.479254+0000 2024-03-25T18:29:55.480 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:29:55.479254+0000 2024-03-25T18:29:55.480 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:55.480 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:55.480 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:55.480 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:29:55.480 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:29:55.480 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:55.481 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.478+0000 7f7b6357b640 1 Processor -- start 2024-03-25T18:29:55.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.479+0000 7f7b6357b640 1 -- start start 2024-03-25T18:29:55.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.479+0000 7f7b6357b640 10 monclient: init 2024-03-25T18:29:55.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.479+0000 7f7b6357b640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:55.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.479+0000 7f7b6357b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:55.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.479+0000 7f7b6357b640 10 monclient(hunting): picked mon.noname-c con 0x7f7b5c0a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:55.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.479+0000 7f7b6357b640 10 monclient(hunting): picked mon.noname-a con 0x7f7b5c13f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:55.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.479+0000 7f7b6357b640 10 monclient(hunting): picked mon.noname-b con 0x7f7b5c1430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:55.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.479+0000 7f7b6357b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5c142520 con 0x7f7b5c13f710 2024-03-25T18:29:55.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.479+0000 7f7b6357b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5c0b28a0 con 0x7f7b5c0a98f0 2024-03-25T18:29:55.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.479+0000 7f7b6357b640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5c0b2a30 con 0x7f7b5c1430c0 2024-03-25T18:29:55.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.479+0000 7f7b6357b640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:55.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.479+0000 7f7b6357b640 10 monclient(hunting): authenticate will time out at 7613.647717s 2024-03-25T18:29:55.487 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.486+0000 7f7b61317640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f7b5c0a98f0 0x7f7b5c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:39950/0 (socket says 172.21.15.99:39950) 2024-03-25T18:29:55.487 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.486+0000 7f7b61317640 1 -- 172.21.15.99:0/4105811225 learned_addr learned my addr 172.21.15.99:0/4105811225 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:55.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.494+0000 7f7b53fff640 1 -- 172.21.15.99:0/4105811225 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409744581 0 0) 0x7f7b5c0b28a0 con 0x7f7b5c0a98f0 2024-03-25T18:29:55.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.494+0000 7f7b53fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.494+0000 7f7b53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.494+0000 7f7b53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.494+0000 7f7b53fff640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:29:55.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.494+0000 7f7b53fff640 1 -- 172.21.15.99:0/4105811225 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b48003520 con 0x7f7b5c0a98f0 2024-03-25T18:29:55.505 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.504+0000 7f7b53fff640 1 -- 172.21.15.99:0/4105811225 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531369658 0 0) 0x7f7b5c0b2a30 con 0x7f7b5c1430c0 2024-03-25T18:29:55.505 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.504+0000 7f7b53fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.504+0000 7f7b53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.504+0000 7f7b53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.504+0000 7f7b53fff640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:29:55.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.504+0000 7f7b53fff640 1 -- 172.21.15.99:0/4105811225 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b5c0b28a0 con 0x7f7b5c1430c0 2024-03-25T18:29:55.507 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 1 -- 172.21.15.99:0/4105811225 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3468711477 0 0) 0x7f7b48003520 con 0x7f7b5c0a98f0 2024-03-25T18:29:55.507 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 1 -- 172.21.15.99:0/4105811225 >> v1:172.21.15.99:6790/0 conn(0x7f7b5c1430c0 legacy=0x7f7b5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.507 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 1 -- 172.21.15.99:0/4105811225 >> v1:172.21.15.99:6789/0 conn(0x7f7b5c13f710 legacy=0x7f7b5c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 1 monclient: found mon.noname-c 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 1 -- 172.21.15.99:0/4105811225 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5c0b44c0 con 0x7f7b5c0a98f0 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 10 monclient: _finish_auth 0 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 1 -- 172.21.15.99:0/4105811225 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f7b58001690 con 0x7f7b5c0a98f0 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b53fff640 10 monclient: dump: 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:55.508 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:55.509 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:55.509 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:55.509 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:55.509 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:55.509 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:55.509 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:55.509 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:55.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b6357b640 5 monclient: authenticate success, global_id 4426 2024-03-25T18:29:55.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.506+0000 7f7b6357b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:55.520 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.518+0000 7f7b53fff640 1 -- 172.21.15.99:0/4105811225 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b580017f0 con 0x7f7b5c0a98f0 2024-03-25T18:29:55.520 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.518+0000 7f7b53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:55.520 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.518+0000 7f7b53fff640 1 -- 172.21.15.99:0/4105811225 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f7b58001c00 con 0x7f7b5c0a98f0 2024-03-25T18:29:55.520 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.518+0000 7f7b53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:55.520 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.518+0000 7f7b53fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:55.520 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.518+0000 7f7b53fff640 10 monclient: dump: 2024-03-25T18:29:55.520 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:55.520 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:55.520 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:55.520 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:55.520 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:55.520 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:55.520 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:55.521 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:55.521 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:55.521 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:55.521 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.518+0000 7f7b6357b640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:55.521 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.518+0000 7f7b6357b640 4 set_mon_vals no callback set 2024-03-25T18:29:55.522 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.519+0000 7f7b6357b640 10 monclient: shutdown 2024-03-25T18:29:55.522 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.519+0000 7f7b6357b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:55.522 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.519+0000 7f7b6357b640 1 -- 172.21.15.99:0/4105811225 >> v1:172.21.15.190:6789/0 conn(0x7f7b5c0a98f0 legacy=0x7f7b5c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.523 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.519+0000 7f7b6357b640 1 -- 172.21.15.99:0/4105811225 shutdown_connections 2024-03-25T18:29:55.523 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.519+0000 7f7b6357b640 1 -- 172.21.15.99:0/4105811225 >> 172.21.15.99:0/4105811225 conn(0x7f7b5c0b7ea0 msgr2=0x7f7b5c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:55.523 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.519+0000 7f7b6357b640 1 -- 172.21.15.99:0/4105811225 shutdown_connections 2024-03-25T18:29:55.523 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.519+0000 7f7b6357b640 1 -- 172.21.15.99:0/4105811225 wait complete. 2024-03-25T18:29:55.523 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.520+0000 7f7b6357b640 10 monclient: build_initial_monmap 2024-03-25T18:29:55.523 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.520+0000 7f7b6357b640 10 monclient: monmap: 2024-03-25T18:29:55.523 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:29:55.523 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:55.523 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:29:55.523 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:29:55.523 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:55.523 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:29:55.523 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:29:55.523 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:29:55.524 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:29:55.524 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:55.524 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.521+0000 7f7b6357b640 1 librados: starting msgr at 2024-03-25T18:29:55.524 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.521+0000 7f7b6357b640 1 librados: starting objecter 2024-03-25T18:29:55.524 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.521+0000 7f7b6357b640 1 Processor -- start 2024-03-25T18:29:55.524 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.521+0000 7f7b6357b640 1 -- start start 2024-03-25T18:29:55.524 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.521+0000 7f7b6357b640 1 librados: setting wanted keys 2024-03-25T18:29:55.524 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.521+0000 7f7b6357b640 1 librados: calling monclient init 2024-03-25T18:29:55.524 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.521+0000 7f7b6357b640 10 monclient: init 2024-03-25T18:29:55.525 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.522+0000 7f7b6357b640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:55.525 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.522+0000 7f7b6357b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:55.525 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.522+0000 7f7b6357b640 10 monclient(hunting): picked mon.noname-a con 0x7f7b5c0a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:29:55.525 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.522+0000 7f7b6357b640 10 monclient(hunting): picked mon.noname-b con 0x7f7b5c13f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:29:55.525 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.522+0000 7f7b6357b640 10 monclient(hunting): picked mon.noname-c con 0x7f7b5c1430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:29:55.525 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.522+0000 7f7b6357b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5c132ba0 con 0x7f7b5c0a98f0 2024-03-25T18:29:55.525 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.522+0000 7f7b6357b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5c138150 con 0x7f7b5c13f710 2024-03-25T18:29:55.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.522+0000 7f7b6357b640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5c139510 con 0x7f7b5c1430c0 2024-03-25T18:29:55.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.522+0000 7f7b6357b640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:55.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.522+0000 7f7b6357b640 10 monclient(hunting): authenticate will time out at 7613.690274s 2024-03-25T18:29:55.531 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.530+0000 7f7b61b18640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f7b5c1430c0 0x7f7b5c1e27a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:50938/0 (socket says 172.21.15.99:50938) 2024-03-25T18:29:55.531 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.530+0000 7f7b61b18640 1 -- 172.21.15.99:0/534448144 learned_addr learned my addr 172.21.15.99:0/534448144 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:29:55.531 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.530+0000 7f7b51ffb640 10 client.?.objecter ms_handle_connect 0x7f7b5c1430c0 2024-03-25T18:29:55.531 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.530+0000 7f7b51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:55.539 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.538+0000 7f7b51ffb640 10 client.?.objecter ms_handle_connect 0x7f7b5c13f710 2024-03-25T18:29:55.539 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.538+0000 7f7b51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:55.542 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.541+0000 7f7b51ffb640 10 client.?.objecter ms_handle_connect 0x7f7b5c0a98f0 2024-03-25T18:29:55.542 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.541+0000 7f7b51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:55.547 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.546+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767103695 0 0) 0x7f7b5c139510 con 0x7f7b5c1430c0 2024-03-25T18:29:55.547 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.546+0000 7f7b51ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.547 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.546+0000 7f7b51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.547 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.546+0000 7f7b51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.547 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.546+0000 7f7b51ffb640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:29:55.547 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.546+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b40004eb0 con 0x7f7b5c1430c0 2024-03-25T18:29:55.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.551+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071084123 0 0) 0x7f7b5c0b44c0 con 0x7f7b5c0a98f0 2024-03-25T18:29:55.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.551+0000 7f7b51ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.551+0000 7f7b51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.551+0000 7f7b51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.551+0000 7f7b51ffb640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:29:55.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.551+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b5c139510 con 0x7f7b5c0a98f0 2024-03-25T18:29:55.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.553+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133166394 0 0) 0x7f7b5c138150 con 0x7f7b5c13f710 2024-03-25T18:29:55.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.553+0000 7f7b51ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:55.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.553+0000 7f7b51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:55.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.553+0000 7f7b51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:55.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.553+0000 7f7b51ffb640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:29:55.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.553+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b5c0b44c0 con 0x7f7b5c13f710 2024-03-25T18:29:55.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.555+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1431604465 0 0) 0x7f7b40004eb0 con 0x7f7b5c1430c0 2024-03-25T18:29:55.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.555+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b5c138150 con 0x7f7b5c1430c0 2024-03-25T18:29:55.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.555+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f7b54001c90 con 0x7f7b5c1430c0 2024-03-25T18:29:55.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.555+0000 7f7b51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:55.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.555+0000 7f7b51ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:55.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.555+0000 7f7b51ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:55.556 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:55.556 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:55.556 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:55.556 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:55.557 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:55.557 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:55.557 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:55.557 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:55.557 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:55.557 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:55.557 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.555+0000 7f7b51ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:55.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2819432324 0 0) 0x7f7b5c138150 con 0x7f7b5c1430c0 2024-03-25T18:29:55.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 >> v1:172.21.15.190:6789/0 conn(0x7f7b5c13f710 legacy=0x7f7b5c135000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 >> v1:172.21.15.99:6789/0 conn(0x7f7b5c0a98f0 legacy=0x7f7b5c137190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:55.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b51ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:55.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b51ffb640 1 monclient: found mon.c 2024-03-25T18:29:55.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:55.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b51ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:29:55.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5c1397f0 con 0x7f7b5c1430c0 2024-03-25T18:29:55.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b51ffb640 10 monclient: _finish_auth 0 2024-03-25T18:29:55.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b51ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:55.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:55.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b6357b640 5 monclient: authenticate success, global_id 4424 2024-03-25T18:29:55.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b6357b640 10 monclient: _renew_subs 2024-03-25T18:29:55.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b6357b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:29:55.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b6357b640 1 -- 172.21.15.99:0/534448144 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5c138600 con 0x7f7b5c1430c0 2024-03-25T18:29:55.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b6357b640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:55.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b6357b640 10 monclient: _renew_subs 2024-03-25T18:29:55.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b6357b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:29:55.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b6357b640 1 -- 172.21.15.99:0/534448144 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b5c135950 con 0x7f7b5c1430c0 2024-03-25T18:29:55.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.563+0000 7f7b6357b640 1 librados: init done 2024-03-25T18:29:55.567 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.564+0000 7f7b6357b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:55.567 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.564+0000 7f7b6357b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:55.567 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.564+0000 7f7b6357b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:29:55.567 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.564+0000 7f7b6357b640 1 -- 172.21.15.99:0/534448144 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b5c0627b0 con 0x7f7b5c1430c0 2024-03-25T18:29:55.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.577+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b54003970 con 0x7f7b5c1430c0 2024-03-25T18:29:55.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.577+0000 7f7b51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:55.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.577+0000 7f7b537fe640 4 set_mon_vals no callback set 2024-03-25T18:29:55.873 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.872+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f7b54003d40 con 0x7f7b5c1430c0 2024-03-25T18:29:55.873 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.872+0000 7f7b51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:55.873 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.872+0000 7f7b51ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:55.873 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.872+0000 7f7b51ffb640 10 monclient: dump: 2024-03-25T18:29:55.873 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:29:55.873 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:29:55.873 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:55.873 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:29:55.873 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:55.874 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:29:55.874 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:29:55.874 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:29:55.874 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:29:55.874 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:29:55.874 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.872+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f7b54003f20 con 0x7f7b5c1430c0 2024-03-25T18:29:55.874 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.873+0000 7f7b61317640 10 monclient: get_auth_request con 0x7f7b4006ddf0 auth_method 0 2024-03-25T18:29:55.875 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.873+0000 7f7b51ffb640 10 client.4424.objecter ms_handle_connect 0x7f7b4006ddf0 2024-03-25T18:29:55.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.881+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f7b54054b70 con 0x7f7b5c1430c0 2024-03-25T18:29:55.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.881+0000 7f7b51ffb640 10 client.4424.objecter ms_dispatch 0x7f7b5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:55.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.881+0000 7f7b51ffb640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:55.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.881+0000 7f7b51ffb640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:55.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.881+0000 7f7b51ffb640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-25T18:29:55.883 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.881+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b54050680 con 0x7f7b5c1430c0 2024-03-25T18:29:55.883 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.881+0000 7f7b51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:55.883 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:55.881+0000 7f7b51ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:56.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.108+0000 7f7b6357b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:56.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.108+0000 7f7b6357b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:56.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.108+0000 7f7b6357b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:29:56.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.108+0000 7f7b6357b640 1 -- 172.21.15.99:0/534448144 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b5c13dcf0 con 0x7f7b5c1430c0 2024-03-25T18:29:56.148 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.147+0000 7f7b51ffb640 1 -- 172.21.15.99:0/534448144 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f7b540525b0 con 0x7f7b5c1430c0 2024-03-25T18:29:56.148 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.147+0000 7f7b51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:56.148 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.147+0000 7f7b51ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:56.148 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:29:56.149 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:56.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.149+0000 7f7b6357b640 10 librados: watch_flush enter 2024-03-25T18:29:56.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.149+0000 7f7b6357b640 10 librados: watch_flush exit 2024-03-25T18:29:56.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.149+0000 7f7b6357b640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-25T18:29:56.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.149+0000 7f7b6357b640 10 client.4424.objecter successfully canceled tick 2024-03-25T18:29:56.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.149+0000 7f7b6357b640 1 -- 172.21.15.99:0/534448144 >> v1:172.21.15.190:6800/4220409699 conn(0x7f7b4006ddf0 legacy=0x7f7b400702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:56.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.149+0000 7f7b6357b640 10 monclient: shutdown 2024-03-25T18:29:56.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.149+0000 7f7b6357b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:56.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.149+0000 7f7b6357b640 1 -- 172.21.15.99:0/534448144 >> v1:172.21.15.99:6790/0 conn(0x7f7b5c1430c0 legacy=0x7f7b5c1e27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:56.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.150+0000 7f7b6357b640 1 -- 172.21.15.99:0/534448144 shutdown_connections 2024-03-25T18:29:56.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.150+0000 7f7b6357b640 1 -- 172.21.15.99:0/534448144 >> 172.21.15.99:0/534448144 conn(0x7f7b5c0b7ea0 msgr2=0x7f7b5c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:56.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.150+0000 7f7b6357b640 1 -- 172.21.15.99:0/534448144 shutdown_connections 2024-03-25T18:29:56.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.150+0000 7f7b6357b640 1 -- 172.21.15.99:0/534448144 wait complete. 2024-03-25T18:29:56.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:29:56.150+0000 7f7b6357b640 1 librados: shutdown 2024-03-25T18:29:56.163 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:02.969 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:03.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.082+0000 7efd6022e640 10 monclient: get_monmap_and_config 2024-03-25T18:30:03.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.083+0000 7efd6022e640 10 monclient: build_initial_monmap 2024-03-25T18:30:03.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.083+0000 7efd6022e640 10 monclient: monmap: 2024-03-25T18:30:03.084 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:03.084 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:03.084 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:30:03.084050+0000 2024-03-25T18:30:03.084 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:30:03.084050+0000 2024-03-25T18:30:03.084 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:03.084 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:03.085 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:03.085 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:30:03.085 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:30:03.085 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:03.086 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.083+0000 7efd6022e640 1 Processor -- start 2024-03-25T18:30:03.086 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.083+0000 7efd6022e640 1 -- start start 2024-03-25T18:30:03.086 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.083+0000 7efd6022e640 10 monclient: init 2024-03-25T18:30:03.086 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.084+0000 7efd6022e640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:03.086 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.084+0000 7efd6022e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:03.086 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.084+0000 7efd6022e640 10 monclient(hunting): picked mon.noname-c con 0x7efd580b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:03.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.084+0000 7efd6022e640 10 monclient(hunting): picked mon.noname-b con 0x7efd580b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:03.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.084+0000 7efd6022e640 10 monclient(hunting): picked mon.noname-a con 0x7efd5814be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:03.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.084+0000 7efd6022e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd580af940 con 0x7efd5814be90 2024-03-25T18:30:03.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.084+0000 7efd6022e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd580b0a70 con 0x7efd580b4d60 2024-03-25T18:30:03.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.084+0000 7efd6022e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd580b0d50 con 0x7efd580b1730 2024-03-25T18:30:03.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.084+0000 7efd6022e640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:03.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.084+0000 7efd6022e640 10 monclient(hunting): authenticate will time out at 7621.252717s 2024-03-25T18:30:03.102 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.101+0000 7efd5dfca640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7efd580b4d60 0x7efd5813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:44796/0 (socket says 172.21.15.99:44796) 2024-03-25T18:30:03.102 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.101+0000 7efd5dfca640 1 -- 172.21.15.99:0/2844997992 learned_addr learned my addr 172.21.15.99:0/2844997992 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:03.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.112+0000 7efd5cfc8640 1 -- 172.21.15.99:0/2844997992 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056639952 0 0) 0x7efd580b0a70 con 0x7efd580b4d60 2024-03-25T18:30:03.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.112+0000 7efd5cfc8640 10 monclient(hunting): _negotiate 2024-03-25T18:30:03.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.112+0000 7efd5cfc8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:03.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.112+0000 7efd5cfc8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:03.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.112+0000 7efd5cfc8640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:30:03.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.112+0000 7efd5cfc8640 1 -- 172.21.15.99:0/2844997992 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd44003520 con 0x7efd580b4d60 2024-03-25T18:30:03.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.115+0000 7efd5cfc8640 1 -- 172.21.15.99:0/2844997992 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550977084 0 0) 0x7efd580b0d50 con 0x7efd580b1730 2024-03-25T18:30:03.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.115+0000 7efd5cfc8640 10 monclient(hunting): _negotiate 2024-03-25T18:30:03.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.115+0000 7efd5cfc8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:03.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.115+0000 7efd5cfc8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:03.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.115+0000 7efd5cfc8640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:30:03.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.115+0000 7efd5cfc8640 1 -- 172.21.15.99:0/2844997992 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd580b0a70 con 0x7efd580b1730 2024-03-25T18:30:03.118 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 1 -- 172.21.15.99:0/2844997992 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1775394149 0 0) 0x7efd44003520 con 0x7efd580b4d60 2024-03-25T18:30:03.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 1 -- 172.21.15.99:0/2844997992 >> v1:172.21.15.99:6790/0 conn(0x7efd580b1730 legacy=0x7efd580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:03.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 1 -- 172.21.15.99:0/2844997992 >> v1:172.21.15.99:6789/0 conn(0x7efd5814be90 legacy=0x7efd5814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:30:03.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 10 monclient: _finish_hunting 0 2024-03-25T18:30:03.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 1 monclient: found mon.noname-c 2024-03-25T18:30:03.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:03.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:30:03.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 1 -- 172.21.15.99:0/2844997992 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd580b1030 con 0x7efd580b4d60 2024-03-25T18:30:03.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 10 monclient: _finish_auth 0 2024-03-25T18:30:03.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 10 monclient: _check_auth_tickets 2024-03-25T18:30:03.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:03.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 1 -- 172.21.15.99:0/2844997992 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7efd54001690 con 0x7efd580b4d60 2024-03-25T18:30:03.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:03.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:03.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd5cfc8640 10 monclient: dump: 2024-03-25T18:30:03.120 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:03.120 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:03.120 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:03.120 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:03.120 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:03.120 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:03.120 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:03.120 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:03.120 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:03.120 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:03.120 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd6022e640 5 monclient: authenticate success, global_id 4432 2024-03-25T18:30:03.120 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.117+0000 7efd6022e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:03.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.133+0000 7efd5cfc8640 1 -- 172.21.15.99:0/2844997992 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd54001870 con 0x7efd580b4d60 2024-03-25T18:30:03.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.133+0000 7efd5cfc8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:03.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.133+0000 7efd5cfc8640 1 -- 172.21.15.99:0/2844997992 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7efd54001c80 con 0x7efd580b4d60 2024-03-25T18:30:03.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.133+0000 7efd6022e640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:03.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.133+0000 7efd6022e640 4 set_mon_vals no callback set 2024-03-25T18:30:03.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.133+0000 7efd5cfc8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:03.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.133+0000 7efd5cfc8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:03.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.133+0000 7efd5cfc8640 10 monclient: dump: 2024-03-25T18:30:03.135 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:03.135 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:03.135 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:03.135 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:03.135 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:03.135 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:03.135 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:03.135 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:03.135 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:03.135 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:03.137 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.134+0000 7efd6022e640 10 monclient: shutdown 2024-03-25T18:30:03.137 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.134+0000 7efd6022e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:03.137 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.134+0000 7efd6022e640 1 -- 172.21.15.99:0/2844997992 >> v1:172.21.15.190:6789/0 conn(0x7efd580b4d60 legacy=0x7efd5813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:03.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.134+0000 7efd6022e640 1 -- 172.21.15.99:0/2844997992 shutdown_connections 2024-03-25T18:30:03.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.134+0000 7efd6022e640 1 -- 172.21.15.99:0/2844997992 >> 172.21.15.99:0/2844997992 conn(0x7efd580617a0 msgr2=0x7efd58061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:03.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.134+0000 7efd6022e640 1 -- 172.21.15.99:0/2844997992 shutdown_connections 2024-03-25T18:30:03.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.134+0000 7efd6022e640 1 -- 172.21.15.99:0/2844997992 wait complete. 2024-03-25T18:30:03.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.134+0000 7efd6022e640 10 monclient: build_initial_monmap 2024-03-25T18:30:03.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.134+0000 7efd6022e640 10 monclient: monmap: 2024-03-25T18:30:03.138 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:03.138 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:03.138 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:30:03.138 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:30:03.138 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:03.138 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.135+0000 7efd6022e640 1 librados: starting msgr at 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.135+0000 7efd6022e640 1 librados: starting objecter 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.135+0000 7efd6022e640 1 Processor -- start 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.135+0000 7efd6022e640 1 -- start start 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.135+0000 7efd6022e640 1 librados: setting wanted keys 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.135+0000 7efd6022e640 1 librados: calling monclient init 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.135+0000 7efd6022e640 10 monclient: init 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.135+0000 7efd6022e640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.135+0000 7efd6022e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.135+0000 7efd6022e640 10 monclient(hunting): picked mon.noname-c con 0x7efd580b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:03.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.135+0000 7efd6022e640 10 monclient(hunting): picked mon.noname-b con 0x7efd580b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:03.140 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.135+0000 7efd6022e640 10 monclient(hunting): picked mon.noname-a con 0x7efd5814be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:03.140 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.136+0000 7efd6022e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd581398e0 con 0x7efd5814be90 2024-03-25T18:30:03.140 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.136+0000 7efd6022e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5813a780 con 0x7efd580b4d60 2024-03-25T18:30:03.140 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.136+0000 7efd6022e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5813af90 con 0x7efd580b1730 2024-03-25T18:30:03.140 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.136+0000 7efd6022e640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:03.140 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.136+0000 7efd6022e640 10 monclient(hunting): authenticate will time out at 7621.303995s 2024-03-25T18:30:03.147 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.146+0000 7efd5e7cb640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7efd5814be90 0x7efd5813bb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:59184/0 (socket says 172.21.15.99:59184) 2024-03-25T18:30:03.147 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.146+0000 7efd5e7cb640 1 -- 172.21.15.99:0/2266245372 learned_addr learned my addr 172.21.15.99:0/2266245372 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:03.153 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.152+0000 7efd4e7fc640 10 client.?.objecter ms_handle_connect 0x7efd580b4d60 2024-03-25T18:30:03.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.153+0000 7efd4e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:03.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.153+0000 7efd4e7fc640 10 client.?.objecter ms_handle_connect 0x7efd5814be90 2024-03-25T18:30:03.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.153+0000 7efd4e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:03.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.154+0000 7efd4e7fc640 10 client.?.objecter ms_handle_connect 0x7efd580b1730 2024-03-25T18:30:03.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.154+0000 7efd4e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:03.162 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.161+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746105938 0 0) 0x7efd5813a780 con 0x7efd580b4d60 2024-03-25T18:30:03.163 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.161+0000 7efd4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:03.163 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.161+0000 7efd4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:03.163 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.161+0000 7efd4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:03.163 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.161+0000 7efd4e7fc640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:30:03.163 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.161+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c004eb0 con 0x7efd580b4d60 2024-03-25T18:30:03.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.166+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59087949 0 0) 0x7efd5813af90 con 0x7efd580b1730 2024-03-25T18:30:03.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.166+0000 7efd4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:03.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.166+0000 7efd4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:03.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.166+0000 7efd4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:03.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.166+0000 7efd4e7fc640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:30:03.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.166+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd5813a780 con 0x7efd580b1730 2024-03-25T18:30:03.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.166+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886788373 0 0) 0x7efd581398e0 con 0x7efd5814be90 2024-03-25T18:30:03.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.166+0000 7efd4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:03.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.166+0000 7efd4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:03.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.166+0000 7efd4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:03.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.166+0000 7efd4e7fc640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:30:03.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.166+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd5813af90 con 0x7efd5814be90 2024-03-25T18:30:03.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.171+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2094422859 0 0) 0x7efd3c004eb0 con 0x7efd580b4d60 2024-03-25T18:30:03.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.171+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd581398e0 con 0x7efd580b4d60 2024-03-25T18:30:03.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.171+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7efd48001930 con 0x7efd580b4d60 2024-03-25T18:30:03.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.171+0000 7efd4e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:03.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.171+0000 7efd4e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:03.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.171+0000 7efd4e7fc640 10 monclient(hunting): dump: 2024-03-25T18:30:03.173 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:03.173 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:03.173 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:03.173 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:03.173 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:03.173 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:03.173 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:03.173 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:03.173 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:03.173 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:03.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.171+0000 7efd4e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:03.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 24236413 0 0) 0x7efd581398e0 con 0x7efd580b4d60 2024-03-25T18:30:03.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 >> v1:172.21.15.99:6790/0 conn(0x7efd580b1730 legacy=0x7efd581391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:03.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 >> v1:172.21.15.99:6789/0 conn(0x7efd5814be90 legacy=0x7efd5813bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:03.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd4e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:03.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd4e7fc640 1 monclient: found mon.b 2024-03-25T18:30:03.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:03.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd4e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:03.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd581434c0 con 0x7efd580b4d60 2024-03-25T18:30:03.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd4e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:03.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd4e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:03.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:03.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd6022e640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:30:03.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd6022e640 10 monclient: _renew_subs 2024-03-25T18:30:03.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd6022e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:03.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd6022e640 1 -- 172.21.15.99:0/2266245372 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd58142bb0 con 0x7efd580b4d60 2024-03-25T18:30:03.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd6022e640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:03.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd6022e640 10 monclient: _renew_subs 2024-03-25T18:30:03.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd6022e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:03.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd6022e640 1 -- 172.21.15.99:0/2266245372 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd581e3410 con 0x7efd580b4d60 2024-03-25T18:30:03.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.181+0000 7efd6022e640 1 librados: init done 2024-03-25T18:30:03.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.182+0000 7efd6022e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:03.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.182+0000 7efd6022e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:03.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.182+0000 7efd6022e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:03.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.182+0000 7efd6022e640 1 -- 172.21.15.99:0/2266245372 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd580627b0 con 0x7efd580b4d60 2024-03-25T18:30:03.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.199+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd48003610 con 0x7efd580b4d60 2024-03-25T18:30:03.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.199+0000 7efd4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:03.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.199+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7efd48003920 con 0x7efd580b4d60 2024-03-25T18:30:03.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.199+0000 7efd4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:03.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.199+0000 7efd4e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:03.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.199+0000 7efd4e7fc640 10 monclient: dump: 2024-03-25T18:30:03.200 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:03.200 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:03.200 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:03.201 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:03.201 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:03.201 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:03.201 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:03.201 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:03.201 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:03.201 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:03.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.199+0000 7efd4ffff640 4 set_mon_vals no callback set 2024-03-25T18:30:03.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.200+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7efd48019930 con 0x7efd580b4d60 2024-03-25T18:30:03.202 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.200+0000 7efd5dfca640 10 monclient: get_auth_request con 0x7efd3c06ddf0 auth_method 0 2024-03-25T18:30:03.202 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.201+0000 7efd4e7fc640 10 client.4435.objecter ms_handle_connect 0x7efd3c06ddf0 2024-03-25T18:30:03.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.202+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7efd48053610 con 0x7efd580b4d60 2024-03-25T18:30:03.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.202+0000 7efd4e7fc640 10 client.4435.objecter ms_dispatch 0x7efd58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:03.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.202+0000 7efd4e7fc640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:03.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.202+0000 7efd4e7fc640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:03.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.202+0000 7efd4e7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-25T18:30:03.464 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.463+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd4804ef50 con 0x7efd580b4d60 2024-03-25T18:30:03.464 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.463+0000 7efd4e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:03.464 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.463+0000 7efd4e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:03.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.711+0000 7efd6022e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:03.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.711+0000 7efd6022e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:03.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.711+0000 7efd6022e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:03.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.711+0000 7efd6022e640 1 -- 172.21.15.99:0/2266245372 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd58069f40 con 0x7efd580b4d60 2024-03-25T18:30:03.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.735+0000 7efd4e7fc640 1 -- 172.21.15.99:0/2266245372 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7efd48051050 con 0x7efd580b4d60 2024-03-25T18:30:03.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.735+0000 7efd4e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:03.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.735+0000 7efd4e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:03.737 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:30:03.737 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:03.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.738+0000 7efd6022e640 10 librados: watch_flush enter 2024-03-25T18:30:03.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.738+0000 7efd6022e640 10 librados: watch_flush exit 2024-03-25T18:30:03.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.738+0000 7efd6022e640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-25T18:30:03.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.738+0000 7efd6022e640 10 client.4435.objecter successfully canceled tick 2024-03-25T18:30:03.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.738+0000 7efd6022e640 1 -- 172.21.15.99:0/2266245372 >> v1:172.21.15.190:6800/4220409699 conn(0x7efd3c06ddf0 legacy=0x7efd3c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:03.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.738+0000 7efd6022e640 10 monclient: shutdown 2024-03-25T18:30:03.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.738+0000 7efd6022e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:03.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.738+0000 7efd6022e640 1 -- 172.21.15.99:0/2266245372 >> v1:172.21.15.190:6789/0 conn(0x7efd580b4d60 legacy=0x7efd5813b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:03.741 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.739+0000 7efd6022e640 1 -- 172.21.15.99:0/2266245372 shutdown_connections 2024-03-25T18:30:03.741 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.739+0000 7efd6022e640 1 -- 172.21.15.99:0/2266245372 >> 172.21.15.99:0/2266245372 conn(0x7efd580617a0 msgr2=0x7efd58137300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:03.741 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.739+0000 7efd6022e640 1 -- 172.21.15.99:0/2266245372 shutdown_connections 2024-03-25T18:30:03.741 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.739+0000 7efd6022e640 1 -- 172.21.15.99:0/2266245372 wait complete. 2024-03-25T18:30:03.741 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:03.739+0000 7efd6022e640 1 librados: shutdown 2024-03-25T18:30:03.750 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:09.181 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:05:09.774 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:09.194 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:05:09.787 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:09.233 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:05:09.825 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:09.233 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:05:09.825 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:09.234 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:05:09.793 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:09.238 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:05:09.797 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:09.249 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:05:09.841 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:09.278 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:05:09.836 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:09.287 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:05:09.880 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:10.558 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:10.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.670+0000 7ff6c95cf640 10 monclient: get_monmap_and_config 2024-03-25T18:30:10.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.670+0000 7ff6c95cf640 10 monclient: build_initial_monmap 2024-03-25T18:30:10.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.670+0000 7ff6c95cf640 10 monclient: monmap: 2024-03-25T18:30:10.672 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:10.672 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:10.672 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:30:10.671821+0000 2024-03-25T18:30:10.672 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:30:10.671821+0000 2024-03-25T18:30:10.672 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:10.672 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:10.672 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:10.672 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:30:10.673 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:30:10.673 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:10.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.671+0000 7ff6c95cf640 1 Processor -- start 2024-03-25T18:30:10.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.671+0000 7ff6c95cf640 1 -- start start 2024-03-25T18:30:10.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.671+0000 7ff6c95cf640 10 monclient: init 2024-03-25T18:30:10.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.672+0000 7ff6c95cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:10.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.672+0000 7ff6c95cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:10.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.672+0000 7ff6c95cf640 10 monclient(hunting): picked mon.noname-a con 0x7ff6c40a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:10.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.672+0000 7ff6c95cf640 10 monclient(hunting): picked mon.noname-b con 0x7ff6c413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:10.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.672+0000 7ff6c95cf640 10 monclient(hunting): picked mon.noname-c con 0x7ff6c41430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:10.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.672+0000 7ff6c95cf640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c4142520 con 0x7ff6c40a98f0 2024-03-25T18:30:10.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.672+0000 7ff6c95cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c40b28a0 con 0x7ff6c41430c0 2024-03-25T18:30:10.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.672+0000 7ff6c95cf640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c40b2a30 con 0x7ff6c413f710 2024-03-25T18:30:10.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.672+0000 7ff6c95cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:10.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.672+0000 7ff6c95cf640 10 monclient(hunting): authenticate will time out at 7628.840333s 2024-03-25T18:30:10.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.678+0000 7ff6c37fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ff6c41430c0 0x7ff6c414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:60102/0 (socket says 172.21.15.99:60102) 2024-03-25T18:30:10.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.678+0000 7ff6c37fe640 1 -- 172.21.15.99:0/2514248802 learned_addr learned my addr 172.21.15.99:0/2514248802 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:10.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.683+0000 7ff6c27fc640 1 -- 172.21.15.99:0/2514248802 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711785296 0 0) 0x7ff6c40b28a0 con 0x7ff6c41430c0 2024-03-25T18:30:10.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.683+0000 7ff6c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:10.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.683+0000 7ff6c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:10.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.683+0000 7ff6c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:10.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.683+0000 7ff6c27fc640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:30:10.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.683+0000 7ff6c27fc640 1 -- 172.21.15.99:0/2514248802 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6b0003520 con 0x7ff6c41430c0 2024-03-25T18:30:10.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 1 -- 172.21.15.99:0/2514248802 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3382464290 0 0) 0x7ff6b0003520 con 0x7ff6c41430c0 2024-03-25T18:30:10.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 1 -- 172.21.15.99:0/2514248802 >> v1:172.21.15.99:6790/0 conn(0x7ff6c413f710 legacy=0x7ff6c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:10.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 1 -- 172.21.15.99:0/2514248802 >> v1:172.21.15.99:6789/0 conn(0x7ff6c40a98f0 legacy=0x7ff6c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:10.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:10.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 1 monclient: found mon.noname-c 2024-03-25T18:30:10.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:10.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:30:10.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 1 -- 172.21.15.99:0/2514248802 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c40b44c0 con 0x7ff6c41430c0 2024-03-25T18:30:10.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:10.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:10.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:10.704 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 1 -- 172.21.15.99:0/2514248802 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff6bc001a10 con 0x7ff6c41430c0 2024-03-25T18:30:10.704 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:10.704 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:10.704 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c27fc640 10 monclient: dump: 2024-03-25T18:30:10.704 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:10.704 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:10.704 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:10.704 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:10.704 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:10.704 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:10.704 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:10.704 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:10.704 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:10.704 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:10.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c95cf640 5 monclient: authenticate success, global_id 4438 2024-03-25T18:30:10.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.701+0000 7ff6c95cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:10.722 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c27fc640 1 -- 172.21.15.99:0/2514248802 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6bc001b70 con 0x7ff6c41430c0 2024-03-25T18:30:10.722 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:10.722 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c27fc640 1 -- 172.21.15.99:0/2514248802 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff6bc001f80 con 0x7ff6c41430c0 2024-03-25T18:30:10.722 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c95cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:10.722 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c95cf640 4 set_mon_vals no callback set 2024-03-25T18:30:10.722 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:10.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:10.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c27fc640 10 monclient: dump: 2024-03-25T18:30:10.723 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:10.723 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:10.723 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:10.723 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:10.723 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:10.723 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:10.723 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:10.723 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:10.723 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:10.723 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:10.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c95cf640 10 monclient: shutdown 2024-03-25T18:30:10.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c95cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:10.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c95cf640 1 -- 172.21.15.99:0/2514248802 >> v1:172.21.15.190:6789/0 conn(0x7ff6c41430c0 legacy=0x7ff6c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:10.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c95cf640 1 -- 172.21.15.99:0/2514248802 shutdown_connections 2024-03-25T18:30:10.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c95cf640 1 -- 172.21.15.99:0/2514248802 >> 172.21.15.99:0/2514248802 conn(0x7ff6c40b7ea0 msgr2=0x7ff6c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:10.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c95cf640 1 -- 172.21.15.99:0/2514248802 shutdown_connections 2024-03-25T18:30:10.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.721+0000 7ff6c95cf640 1 -- 172.21.15.99:0/2514248802 wait complete. 2024-03-25T18:30:10.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.722+0000 7ff6c95cf640 10 monclient: build_initial_monmap 2024-03-25T18:30:10.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.722+0000 7ff6c95cf640 10 monclient: monmap: 2024-03-25T18:30:10.726 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:10.726 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:10.726 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:30:10.726 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:30:10.726 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:10.726 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:30:10.726 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.722+0000 7ff6c95cf640 1 librados: starting msgr at 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.722+0000 7ff6c95cf640 1 librados: starting objecter 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.722+0000 7ff6c95cf640 1 Processor -- start 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.723+0000 7ff6c95cf640 1 -- start start 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.723+0000 7ff6c95cf640 1 librados: setting wanted keys 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.723+0000 7ff6c95cf640 1 librados: calling monclient init 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.723+0000 7ff6c95cf640 10 monclient: init 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.723+0000 7ff6c95cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.723+0000 7ff6c95cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.723+0000 7ff6c95cf640 10 monclient(hunting): picked mon.noname-a con 0x7ff6c40a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.723+0000 7ff6c95cf640 10 monclient(hunting): picked mon.noname-b con 0x7ff6c413f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:10.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.723+0000 7ff6c95cf640 10 monclient(hunting): picked mon.noname-c con 0x7ff6c41430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:10.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.723+0000 7ff6c95cf640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c41d7ba0 con 0x7ff6c40a98f0 2024-03-25T18:30:10.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.723+0000 7ff6c95cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c4138330 con 0x7ff6c413f710 2024-03-25T18:30:10.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.723+0000 7ff6c95cf640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c40ab950 con 0x7ff6c41430c0 2024-03-25T18:30:10.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.723+0000 7ff6c95cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:10.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.723+0000 7ff6c95cf640 10 monclient(hunting): authenticate will time out at 7628.891490s 2024-03-25T18:30:10.738 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.736+0000 7ff6c37fe640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7ff6c41430c0 0x7ff6c40b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:54564/0 (socket says 172.21.15.99:54564) 2024-03-25T18:30:10.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.736+0000 7ff6c37fe640 1 -- 172.21.15.99:0/3296842224 learned_addr learned my addr 172.21.15.99:0/3296842224 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:10.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.737+0000 7ff6bb7fe640 10 client.?.objecter ms_handle_connect 0x7ff6c41430c0 2024-03-25T18:30:10.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.737+0000 7ff6bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:10.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.744+0000 7ff6bb7fe640 10 client.?.objecter ms_handle_connect 0x7ff6c40a98f0 2024-03-25T18:30:10.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.744+0000 7ff6bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:10.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.747+0000 7ff6bb7fe640 10 client.?.objecter ms_handle_connect 0x7ff6c413f710 2024-03-25T18:30:10.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.747+0000 7ff6bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:10.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.753+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334586605 0 0) 0x7ff6c40ab950 con 0x7ff6c41430c0 2024-03-25T18:30:10.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.753+0000 7ff6bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:10.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.753+0000 7ff6bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:10.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.753+0000 7ff6bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:10.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.753+0000 7ff6bb7fe640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:30:10.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.753+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6a4004eb0 con 0x7ff6c41430c0 2024-03-25T18:30:10.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.758+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34519826 0 0) 0x7ff6c4138330 con 0x7ff6c413f710 2024-03-25T18:30:10.760 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.758+0000 7ff6bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:10.760 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.758+0000 7ff6bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:10.760 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.758+0000 7ff6bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:10.760 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.758+0000 7ff6bb7fe640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:30:10.760 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.758+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c40ab950 con 0x7ff6c413f710 2024-03-25T18:30:10.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.762+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919979473 0 0) 0x7ff6c41d7ba0 con 0x7ff6c40a98f0 2024-03-25T18:30:10.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.762+0000 7ff6bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:10.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.762+0000 7ff6bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:10.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.762+0000 7ff6bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:10.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.762+0000 7ff6bb7fe640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:30:10.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.762+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c4138330 con 0x7ff6c40a98f0 2024-03-25T18:30:10.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.770+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1030647984 0 0) 0x7ff6c40ab950 con 0x7ff6c413f710 2024-03-25T18:30:10.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.770+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6c41d7ba0 con 0x7ff6c413f710 2024-03-25T18:30:10.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.770+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff6ac0018b0 con 0x7ff6c413f710 2024-03-25T18:30:10.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.770+0000 7ff6bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:10.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.770+0000 7ff6bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:10.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.770+0000 7ff6bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:30:10.772 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:10.772 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:10.773 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:10.773 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:10.773 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:10.773 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:10.773 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:10.773 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:10.773 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:10.773 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:10.773 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.770+0000 7ff6bb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:10.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.775+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3069499342 0 0) 0x7ff6c4138330 con 0x7ff6c40a98f0 2024-03-25T18:30:10.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.775+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6c40ab950 con 0x7ff6c40a98f0 2024-03-25T18:30:10.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.775+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff6b4001830 con 0x7ff6c40a98f0 2024-03-25T18:30:10.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.775+0000 7ff6bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:10.777 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.775+0000 7ff6bb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:10.777 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.775+0000 7ff6bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:30:10.777 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:10.777 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:10.777 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:10.777 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:10.777 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:10.777 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:10.777 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:10.777 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:10.777 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:10.778 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:10.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 531638859 0 0) 0x7ff6c41d7ba0 con 0x7ff6c413f710 2024-03-25T18:30:10.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 >> v1:172.21.15.99:6790/0 conn(0x7ff6c41430c0 legacy=0x7ff6c40b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:10.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 >> v1:172.21.15.99:6789/0 conn(0x7ff6c40a98f0 legacy=0x7ff6c41378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:10.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6bb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:30:10.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6bb7fe640 1 monclient: found mon.b 2024-03-25T18:30:10.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:10.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6bb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:10.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c40b1320 con 0x7ff6c413f710 2024-03-25T18:30:10.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6bb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:30:10.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6bb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:30:10.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:10.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6c95cf640 5 monclient: authenticate success, global_id 4441 2024-03-25T18:30:10.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6c95cf640 10 monclient: _renew_subs 2024-03-25T18:30:10.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6c95cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:10.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6c95cf640 1 -- 172.21.15.99:0/3296842224 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c40b15d0 con 0x7ff6c413f710 2024-03-25T18:30:10.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6c95cf640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:10.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6c95cf640 10 monclient: _renew_subs 2024-03-25T18:30:10.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6c95cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:10.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6c95cf640 1 -- 172.21.15.99:0/3296842224 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c40b09d0 con 0x7ff6c413f710 2024-03-25T18:30:10.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.780+0000 7ff6c95cf640 1 librados: init done 2024-03-25T18:30:10.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.781+0000 7ff6c95cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:10.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.781+0000 7ff6c95cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:10.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.781+0000 7ff6c95cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:10.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:10.781+0000 7ff6c95cf640 1 -- 172.21.15.99:0/3296842224 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c4138190 con 0x7ff6c413f710 2024-03-25T18:30:11.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.055+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6ac0035f0 con 0x7ff6c413f710 2024-03-25T18:30:11.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.055+0000 7ff6bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:11.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.055+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff6ac0038a0 con 0x7ff6c413f710 2024-03-25T18:30:11.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.055+0000 7ff6bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:11.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.055+0000 7ff6bb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:11.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.055+0000 7ff6bb7fe640 10 monclient: dump: 2024-03-25T18:30:11.057 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:11.057 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:11.057 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:11.057 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:11.057 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:11.057 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:11.057 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:11.057 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:11.058 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:11.058 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:11.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.055+0000 7ff6c1ffb640 4 set_mon_vals no callback set 2024-03-25T18:30:11.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.056+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7ff6ac003a80 con 0x7ff6c413f710 2024-03-25T18:30:11.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.057+0000 7ff6c2ffd640 10 monclient: get_auth_request con 0x7ff6a406df30 auth_method 0 2024-03-25T18:30:11.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.057+0000 7ff6bb7fe640 10 client.4441.objecter ms_handle_connect 0x7ff6a406df30 2024-03-25T18:30:11.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.060+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7ff6ac0534d0 con 0x7ff6c413f710 2024-03-25T18:30:11.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.060+0000 7ff6bb7fe640 10 client.4441.objecter ms_dispatch 0x7ff6c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:11.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.060+0000 7ff6bb7fe640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:11.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.060+0000 7ff6bb7fe640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:11.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.060+0000 7ff6bb7fe640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-25T18:30:11.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.060+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6ac04efe0 con 0x7ff6c413f710 2024-03-25T18:30:11.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.060+0000 7ff6bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:11.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.060+0000 7ff6bb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:11.316 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.315+0000 7ff6c95cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:11.316 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.315+0000 7ff6c95cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:11.316 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.315+0000 7ff6c95cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:11.316 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.315+0000 7ff6c95cf640 1 -- 172.21.15.99:0/3296842224 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6c40627b0 con 0x7ff6c413f710 2024-03-25T18:30:11.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.347+0000 7ff6bb7fe640 1 -- 172.21.15.99:0/3296842224 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7ff6ac050f10 con 0x7ff6c413f710 2024-03-25T18:30:11.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.347+0000 7ff6bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:11.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.347+0000 7ff6bb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:11.348 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:30:11.349 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:11.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.350+0000 7ff6c95cf640 10 librados: watch_flush enter 2024-03-25T18:30:11.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.350+0000 7ff6c95cf640 10 librados: watch_flush exit 2024-03-25T18:30:11.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.350+0000 7ff6c95cf640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-25T18:30:11.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.350+0000 7ff6c95cf640 10 client.4441.objecter successfully canceled tick 2024-03-25T18:30:11.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.350+0000 7ff6c95cf640 1 -- 172.21.15.99:0/3296842224 >> v1:172.21.15.190:6800/4220409699 conn(0x7ff6a406df30 legacy=0x7ff6a40703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:11.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.350+0000 7ff6c95cf640 10 monclient: shutdown 2024-03-25T18:30:11.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.350+0000 7ff6c95cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:11.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.350+0000 7ff6c95cf640 1 -- 172.21.15.99:0/3296842224 >> v1:172.21.15.190:6789/0 conn(0x7ff6c413f710 legacy=0x7ff6c41d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:11.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.350+0000 7ff6c95cf640 1 -- 172.21.15.99:0/3296842224 shutdown_connections 2024-03-25T18:30:11.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.350+0000 7ff6c95cf640 1 -- 172.21.15.99:0/3296842224 >> 172.21.15.99:0/3296842224 conn(0x7ff6c40b7ea0 msgr2=0x7ff6c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:11.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.350+0000 7ff6c95cf640 1 -- 172.21.15.99:0/3296842224 shutdown_connections 2024-03-25T18:30:11.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.350+0000 7ff6c95cf640 1 -- 172.21.15.99:0/3296842224 wait complete. 2024-03-25T18:30:11.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:11.350+0000 7ff6c95cf640 1 librados: shutdown 2024-03-25T18:30:11.364 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:18.172 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:18.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.286+0000 7f764b4dd640 10 monclient: get_monmap_and_config 2024-03-25T18:30:18.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.286+0000 7f764b4dd640 10 monclient: build_initial_monmap 2024-03-25T18:30:18.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.286+0000 7f764b4dd640 10 monclient: monmap: 2024-03-25T18:30:18.287 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:18.287 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:18.287 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:30:18.287199+0000 2024-03-25T18:30:18.287 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:30:18.287199+0000 2024-03-25T18:30:18.288 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:18.288 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:18.288 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:18.288 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:30:18.288 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:30:18.288 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:18.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.286+0000 7f764b4dd640 1 Processor -- start 2024-03-25T18:30:18.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.286+0000 7f764b4dd640 1 -- start start 2024-03-25T18:30:18.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.286+0000 7f764b4dd640 10 monclient: init 2024-03-25T18:30:18.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.287+0000 7f764b4dd640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:18.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.287+0000 7f764b4dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:18.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.287+0000 7f764b4dd640 10 monclient(hunting): picked mon.noname-c con 0x7f76440705b0 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:18.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.287+0000 7f764b4dd640 10 monclient(hunting): picked mon.noname-a con 0x7f76441402d0 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:18.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.287+0000 7f764b4dd640 10 monclient(hunting): picked mon.noname-b con 0x7f7644149ae0 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:18.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.287+0000 7f764b4dd640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76440718c0 con 0x7f76441402d0 2024-03-25T18:30:18.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.287+0000 7f764b4dd640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76440714c0 con 0x7f76440705b0 2024-03-25T18:30:18.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.287+0000 7f764b4dd640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7644072b50 con 0x7f7644149ae0 2024-03-25T18:30:18.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.287+0000 7f764b4dd640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:18.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.287+0000 7f764b4dd640 10 monclient(hunting): authenticate will time out at 7636.455421s 2024-03-25T18:30:18.298 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.297+0000 7f7648a78640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f76441402d0 0x7f7644140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:59194/0 (socket says 172.21.15.99:59194) 2024-03-25T18:30:18.298 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.297+0000 7f7648a78640 1 -- 172.21.15.99:0/2565777181 learned_addr learned my addr 172.21.15.99:0/2565777181 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:18.317 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.316+0000 7f763bfff640 1 -- 172.21.15.99:0/2565777181 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707331902 0 0) 0x7f7644072b50 con 0x7f7644149ae0 2024-03-25T18:30:18.317 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.316+0000 7f763bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:18.317 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.316+0000 7f763bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:18.317 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.316+0000 7f763bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:18.317 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.316+0000 7f763bfff640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:30:18.317 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.316+0000 7f763bfff640 1 -- 172.21.15.99:0/2565777181 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7630003520 con 0x7f7644149ae0 2024-03-25T18:30:18.324 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.323+0000 7f763bfff640 1 -- 172.21.15.99:0/2565777181 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096324296 0 0) 0x7f76440718c0 con 0x7f76441402d0 2024-03-25T18:30:18.324 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.323+0000 7f763bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:18.324 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.323+0000 7f763bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:18.324 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.323+0000 7f763bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:18.324 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.323+0000 7f763bfff640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:30:18.324 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.323+0000 7f763bfff640 1 -- 172.21.15.99:0/2565777181 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7644072b50 con 0x7f76441402d0 2024-03-25T18:30:18.325 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.323+0000 7f763bfff640 1 -- 172.21.15.99:0/2565777181 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015567177 0 0) 0x7f76440714c0 con 0x7f76440705b0 2024-03-25T18:30:18.325 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.323+0000 7f763bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:18.325 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.323+0000 7f763bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:18.325 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.323+0000 7f763bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:18.325 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.323+0000 7f763bfff640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:30:18.325 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.323+0000 7f763bfff640 1 -- 172.21.15.99:0/2565777181 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76440718c0 con 0x7f76440705b0 2024-03-25T18:30:18.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.335+0000 7f763bfff640 1 -- 172.21.15.99:0/2565777181 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1924027996 0 0) 0x7f76440718c0 con 0x7f76440705b0 2024-03-25T18:30:18.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.335+0000 7f763bfff640 1 -- 172.21.15.99:0/2565777181 >> v1:172.21.15.99:6790/0 conn(0x7f7644149ae0 legacy=0x7f764414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:18.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.335+0000 7f763bfff640 1 -- 172.21.15.99:0/2565777181 >> v1:172.21.15.99:6789/0 conn(0x7f76441402d0 legacy=0x7f7644140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:18.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.335+0000 7f763bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:18.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.335+0000 7f763bfff640 1 monclient: found mon.noname-c 2024-03-25T18:30:18.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.335+0000 7f763bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:18.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.335+0000 7f763bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:30:18.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.335+0000 7f763bfff640 1 -- 172.21.15.99:0/2565777181 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764413dd10 con 0x7f76440705b0 2024-03-25T18:30:18.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.335+0000 7f763bfff640 10 monclient: _finish_auth 0 2024-03-25T18:30:18.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.335+0000 7f763bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:18.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.335+0000 7f763bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:18.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.335+0000 7f763bfff640 1 -- 172.21.15.99:0/2565777181 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f76400015b0 con 0x7f76440705b0 2024-03-25T18:30:18.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.335+0000 7f764b4dd640 5 monclient: authenticate success, global_id 4444 2024-03-25T18:30:18.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.336+0000 7f764b4dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:18.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.336+0000 7f763bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:18.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.336+0000 7f763bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:18.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.336+0000 7f763bfff640 10 monclient: dump: 2024-03-25T18:30:18.338 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:18.338 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:18.339 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:18.339 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:18.339 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:18.339 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:18.339 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:18.339 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:18.339 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:18.339 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:18.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.336+0000 7f764b4dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:18.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.348+0000 7f763bfff640 1 -- 172.21.15.99:0/2565777181 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7640001710 con 0x7f76440705b0 2024-03-25T18:30:18.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.348+0000 7f763bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:18.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.348+0000 7f763bfff640 1 -- 172.21.15.99:0/2565777181 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f7640001b60 con 0x7f76440705b0 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.348+0000 7f764b4dd640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.348+0000 7f764b4dd640 4 set_mon_vals no callback set 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.348+0000 7f763bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.348+0000 7f763bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.348+0000 7f763bfff640 10 monclient: dump: 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:18.351 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:18.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.349+0000 7f764b4dd640 10 monclient: shutdown 2024-03-25T18:30:18.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.349+0000 7f764b4dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:18.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.349+0000 7f764b4dd640 1 -- 172.21.15.99:0/2565777181 >> v1:172.21.15.190:6789/0 conn(0x7f76440705b0 legacy=0x7f764413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:18.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.349+0000 7f764b4dd640 1 -- 172.21.15.99:0/2565777181 shutdown_connections 2024-03-25T18:30:18.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.349+0000 7f764b4dd640 1 -- 172.21.15.99:0/2565777181 >> 172.21.15.99:0/2565777181 conn(0x7f76440b5d10 msgr2=0x7f76440b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:18.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.349+0000 7f764b4dd640 1 -- 172.21.15.99:0/2565777181 shutdown_connections 2024-03-25T18:30:18.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.349+0000 7f764b4dd640 1 -- 172.21.15.99:0/2565777181 wait complete. 2024-03-25T18:30:18.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.350+0000 7f764b4dd640 10 monclient: build_initial_monmap 2024-03-25T18:30:18.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.350+0000 7f764b4dd640 10 monclient: monmap: 2024-03-25T18:30:18.354 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:18.354 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:18.354 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:30:18.354 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:30:18.354 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:18.354 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:30:18.354 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:18.354 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:30:18.354 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:30:18.354 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:18.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.350+0000 7f764b4dd640 1 librados: starting msgr at 2024-03-25T18:30:18.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.350+0000 7f764b4dd640 1 librados: starting objecter 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.350+0000 7f764b4dd640 1 Processor -- start 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.350+0000 7f764b4dd640 1 -- start start 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.350+0000 7f764b4dd640 1 librados: setting wanted keys 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.350+0000 7f764b4dd640 1 librados: calling monclient init 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.350+0000 7f764b4dd640 10 monclient: init 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.351+0000 7f764b4dd640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.351+0000 7f764b4dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.351+0000 7f764b4dd640 10 monclient(hunting): picked mon.noname-b con 0x7f76441402d0 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.351+0000 7f764b4dd640 10 monclient(hunting): picked mon.noname-a con 0x7f7644149ae0 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.351+0000 7f764b4dd640 10 monclient(hunting): picked mon.noname-c con 0x7f76441d5b60 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.351+0000 7f764b4dd640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7644137670 con 0x7f7644149ae0 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.351+0000 7f764b4dd640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76441d6080 con 0x7f76441402d0 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.351+0000 7f764b4dd640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7644148ad0 con 0x7f76441d5b60 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.351+0000 7f764b4dd640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:18.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.351+0000 7f764b4dd640 10 monclient(hunting): authenticate will time out at 7636.519393s 2024-03-25T18:30:18.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.361+0000 7f7649a7a640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f76441d5b60 0x7f76441428f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:54774/0 (socket says 172.21.15.99:54774) 2024-03-25T18:30:18.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.361+0000 7f7649a7a640 1 -- 172.21.15.99:0/2713030175 learned_addr learned my addr 172.21.15.99:0/2713030175 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:18.364 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.363+0000 7f7639ffb640 10 client.?.objecter ms_handle_connect 0x7f76441d5b60 2024-03-25T18:30:18.364 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.363+0000 7f7639ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:18.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.366+0000 7f7639ffb640 10 client.?.objecter ms_handle_connect 0x7f7644149ae0 2024-03-25T18:30:18.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.366+0000 7f7639ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:18.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.370+0000 7f7639ffb640 10 client.?.objecter ms_handle_connect 0x7f76441402d0 2024-03-25T18:30:18.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.370+0000 7f7639ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:18.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.381+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740167574 0 0) 0x7f76441d6080 con 0x7f76441402d0 2024-03-25T18:30:18.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.382+0000 7f7639ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:18.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.382+0000 7f7639ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:18.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.382+0000 7f7639ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:18.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.382+0000 7f7639ffb640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:30:18.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.382+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7628004eb0 con 0x7f76441402d0 2024-03-25T18:30:18.385 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.384+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116143870 0 0) 0x7f7644148ad0 con 0x7f76441d5b60 2024-03-25T18:30:18.385 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.384+0000 7f7639ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:18.385 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.384+0000 7f7639ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:18.386 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.384+0000 7f7639ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:18.386 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.384+0000 7f7639ffb640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:30:18.386 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.384+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76441d6080 con 0x7f76441d5b60 2024-03-25T18:30:18.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.386+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167911662 0 0) 0x7f7644137670 con 0x7f7644149ae0 2024-03-25T18:30:18.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.386+0000 7f7639ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:18.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.386+0000 7f7639ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:18.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.386+0000 7f7639ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:18.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.387+0000 7f7639ffb640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:30:18.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.387+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7644148ad0 con 0x7f7644149ae0 2024-03-25T18:30:18.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.399+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2229984704 0 0) 0x7f76441d6080 con 0x7f76441d5b60 2024-03-25T18:30:18.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.399+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7644137670 con 0x7f76441d5b60 2024-03-25T18:30:18.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.399+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f763c001c30 con 0x7f76441d5b60 2024-03-25T18:30:18.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.399+0000 7f7639ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:18.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.399+0000 7f7639ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:18.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.399+0000 7f7639ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:18.401 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:18.401 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:18.401 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:18.401 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:18.401 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:18.401 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:18.401 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:18.401 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:18.401 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:18.402 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:18.402 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.399+0000 7f7639ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:18.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.403+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1484135603 0 0) 0x7f7628004eb0 con 0x7f76441402d0 2024-03-25T18:30:18.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.403+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76441d6080 con 0x7f76441402d0 2024-03-25T18:30:18.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.403+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f7640001a20 con 0x7f76441402d0 2024-03-25T18:30:18.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.403+0000 7f7639ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:18.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.403+0000 7f7639ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:18.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.403+0000 7f7639ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:18.405 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:18.406 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:18.406 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:18.406 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:18.406 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:18.406 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:18.406 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:18.406 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:18.406 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:18.406 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:18.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2813857449 0 0) 0x7f76441d6080 con 0x7f76441402d0 2024-03-25T18:30:18.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 >> v1:172.21.15.99:6790/0 conn(0x7f76441d5b60 legacy=0x7f76441428f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:18.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 >> v1:172.21.15.99:6789/0 conn(0x7f7644149ae0 legacy=0x7f76441421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:18.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f7639ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:30:18.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f7639ffb640 1 monclient: found mon.b 2024-03-25T18:30:18.412 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f7639ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:18.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f7639ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:18.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7644148c60 con 0x7f76441402d0 2024-03-25T18:30:18.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f7639ffb640 10 monclient: _finish_auth 0 2024-03-25T18:30:18.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f7639ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:30:18.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f7639ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:18.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f764b4dd640 5 monclient: authenticate success, global_id 4447 2024-03-25T18:30:18.413 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f764b4dd640 10 monclient: _renew_subs 2024-03-25T18:30:18.414 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f764b4dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:18.414 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f764b4dd640 1 -- 172.21.15.99:0/2713030175 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7644148f10 con 0x7f76441402d0 2024-03-25T18:30:18.414 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f764b4dd640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:18.414 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f764b4dd640 10 monclient: _renew_subs 2024-03-25T18:30:18.414 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f764b4dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:18.414 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f764b4dd640 1 -- 172.21.15.99:0/2713030175 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7644149440 con 0x7f76441402d0 2024-03-25T18:30:18.414 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.410+0000 7f764b4dd640 1 librados: init done 2024-03-25T18:30:18.415 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.411+0000 7f764b4dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:18.415 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.411+0000 7f764b4dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:18.415 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.411+0000 7f764b4dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:18.415 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.411+0000 7f764b4dd640 1 -- 172.21.15.99:0/2713030175 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76440627b0 con 0x7f76441402d0 2024-03-25T18:30:18.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.699+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76400038a0 con 0x7f76441402d0 2024-03-25T18:30:18.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.699+0000 7f7639ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:18.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.699+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f7640003bb0 con 0x7f76441402d0 2024-03-25T18:30:18.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.699+0000 7f7639ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:18.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.699+0000 7f7639ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:18.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.699+0000 7f7639ffb640 10 monclient: dump: 2024-03-25T18:30:18.701 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:18.702 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:18.702 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:18.702 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:18.702 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:18.702 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:18.702 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:18.703 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:18.703 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:18.703 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:18.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.699+0000 7f763b7fe640 4 set_mon_vals no callback set 2024-03-25T18:30:18.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.699+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f7640019b80 con 0x7f76441402d0 2024-03-25T18:30:18.704 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.700+0000 7f7648a78640 10 monclient: get_auth_request con 0x7f762806df60 auth_method 0 2024-03-25T18:30:18.704 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.700+0000 7f7639ffb640 10 client.4447.objecter ms_handle_connect 0x7f762806df60 2024-03-25T18:30:18.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.704+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f7640053650 con 0x7f76441402d0 2024-03-25T18:30:18.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.704+0000 7f7639ffb640 10 client.4447.objecter ms_dispatch 0x7f7644000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:18.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.704+0000 7f7639ffb640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:18.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.704+0000 7f7639ffb640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:18.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.704+0000 7f7639ffb640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-25T18:30:18.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.705+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f764004ef90 con 0x7f76441402d0 2024-03-25T18:30:18.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.705+0000 7f7639ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:18.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.705+0000 7f7639ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:18.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.949+0000 7f764b4dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:18.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.949+0000 7f764b4dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:18.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.949+0000 7f764b4dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:18.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:18.949+0000 7f764b4dd640 1 -- 172.21.15.99:0/2713030175 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7644069f40 con 0x7f76441402d0 2024-03-25T18:30:19.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.035+0000 7f7639ffb640 1 -- 172.21.15.99:0/2713030175 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f7640051090 con 0x7f76441402d0 2024-03-25T18:30:19.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.035+0000 7f7639ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:19.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.035+0000 7f7639ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:19.036 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:30:19.037 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:19.038 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.037+0000 7f764b4dd640 10 librados: watch_flush enter 2024-03-25T18:30:19.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.037+0000 7f764b4dd640 10 librados: watch_flush exit 2024-03-25T18:30:19.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.037+0000 7f764b4dd640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-25T18:30:19.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.037+0000 7f764b4dd640 10 client.4447.objecter successfully canceled tick 2024-03-25T18:30:19.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.037+0000 7f764b4dd640 1 -- 172.21.15.99:0/2713030175 >> v1:172.21.15.190:6800/4220409699 conn(0x7f762806df60 legacy=0x7f7628070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:19.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.037+0000 7f764b4dd640 10 monclient: shutdown 2024-03-25T18:30:19.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.037+0000 7f764b4dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:19.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.037+0000 7f764b4dd640 1 -- 172.21.15.99:0/2713030175 >> v1:172.21.15.190:6789/0 conn(0x7f76441402d0 legacy=0x7f76441d5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:19.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.037+0000 7f764b4dd640 1 -- 172.21.15.99:0/2713030175 shutdown_connections 2024-03-25T18:30:19.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.037+0000 7f764b4dd640 1 -- 172.21.15.99:0/2713030175 >> 172.21.15.99:0/2713030175 conn(0x7f76440b5d10 msgr2=0x7f76440b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:19.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.037+0000 7f764b4dd640 1 -- 172.21.15.99:0/2713030175 shutdown_connections 2024-03-25T18:30:19.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.037+0000 7f764b4dd640 1 -- 172.21.15.99:0/2713030175 wait complete. 2024-03-25T18:30:19.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:19.037+0000 7f764b4dd640 1 librados: shutdown 2024-03-25T18:30:19.051 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:25.858 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:25.981 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.980+0000 7f7e2b2e4640 10 monclient: get_monmap_and_config 2024-03-25T18:30:25.981 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.980+0000 7f7e2b2e4640 10 monclient: build_initial_monmap 2024-03-25T18:30:25.981 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.980+0000 7f7e2b2e4640 10 monclient: monmap: 2024-03-25T18:30:25.981 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:25.981 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:25.981 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:30:25.981129+0000 2024-03-25T18:30:25.981 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:30:25.981129+0000 2024-03-25T18:30:25.981 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:25.981 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:25.982 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:25.982 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:30:25.982 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:30:25.982 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:25.983 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.980+0000 7f7e2b2e4640 1 Processor -- start 2024-03-25T18:30:25.983 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.981+0000 7f7e2b2e4640 1 -- start start 2024-03-25T18:30:25.983 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.981+0000 7f7e2b2e4640 10 monclient: init 2024-03-25T18:30:25.983 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.981+0000 7f7e2b2e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:25.983 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.981+0000 7f7e2b2e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:25.984 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.981+0000 7f7e2b2e4640 10 monclient(hunting): picked mon.noname-c con 0x7f7e240a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:25.984 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.981+0000 7f7e2b2e4640 10 monclient(hunting): picked mon.noname-b con 0x7f7e2413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:25.984 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.981+0000 7f7e2b2e4640 10 monclient(hunting): picked mon.noname-a con 0x7f7e241430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:25.984 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.981+0000 7f7e2b2e4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e24142520 con 0x7f7e241430c0 2024-03-25T18:30:25.984 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.981+0000 7f7e2b2e4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e240b28a0 con 0x7f7e240a98f0 2024-03-25T18:30:25.984 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.982+0000 7f7e2b2e4640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e240b2a30 con 0x7f7e2413f710 2024-03-25T18:30:25.984 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.982+0000 7f7e2b2e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:25.984 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.982+0000 7f7e2b2e4640 10 monclient(hunting): authenticate will time out at 7644.149955s 2024-03-25T18:30:25.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.993+0000 7f7e29881640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f7e241430c0 0x7f7e2414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:59846/0 (socket says 172.21.15.99:59846) 2024-03-25T18:30:25.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:25.993+0000 7f7e29881640 1 -- 172.21.15.99:0/2476472374 learned_addr learned my addr 172.21.15.99:0/2476472374 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:26.007 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.006+0000 7f7e1bfff640 1 -- 172.21.15.99:0/2476472374 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119200222 0 0) 0x7f7e24142520 con 0x7f7e241430c0 2024-03-25T18:30:26.007 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.006+0000 7f7e1bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:26.007 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.006+0000 7f7e1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:26.007 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.006+0000 7f7e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:26.007 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.006+0000 7f7e1bfff640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:30:26.007 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.006+0000 7f7e1bfff640 1 -- 172.21.15.99:0/2476472374 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e10003520 con 0x7f7e241430c0 2024-03-25T18:30:26.015 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.014+0000 7f7e1bfff640 1 -- 172.21.15.99:0/2476472374 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118472921 0 0) 0x7f7e240b28a0 con 0x7f7e240a98f0 2024-03-25T18:30:26.015 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.014+0000 7f7e1bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:26.015 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.014+0000 7f7e1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:26.015 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.014+0000 7f7e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:26.015 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.014+0000 7f7e1bfff640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:30:26.015 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.014+0000 7f7e1bfff640 1 -- 172.21.15.99:0/2476472374 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e24142520 con 0x7f7e240a98f0 2024-03-25T18:30:26.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.014+0000 7f7e1bfff640 1 -- 172.21.15.99:0/2476472374 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2239404962 0 0) 0x7f7e10003520 con 0x7f7e241430c0 2024-03-25T18:30:26.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e1bfff640 1 -- 172.21.15.99:0/2476472374 >> v1:172.21.15.99:6790/0 conn(0x7f7e2413f710 legacy=0x7f7e241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:26.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e1bfff640 1 -- 172.21.15.99:0/2476472374 >> v1:172.21.15.190:6789/0 conn(0x7f7e240a98f0 legacy=0x7f7e240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:26.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e1bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:26.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e1bfff640 1 monclient: found mon.noname-a 2024-03-25T18:30:26.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:26.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e1bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:30:26.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e1bfff640 1 -- 172.21.15.99:0/2476472374 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e240b44c0 con 0x7f7e241430c0 2024-03-25T18:30:26.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e1bfff640 10 monclient: _finish_auth 0 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e1bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e1bfff640 1 -- 172.21.15.99:0/2476472374 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f7e1c001a10 con 0x7f7e241430c0 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e2b2e4640 5 monclient: authenticate success, global_id 4434 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e2b2e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e1bfff640 10 monclient: dump: 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:26.017 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:26.018 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:26.018 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:26.018 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:26.018 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.015+0000 7f7e2b2e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:26.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.029+0000 7f7e1bfff640 1 -- 172.21.15.99:0/2476472374 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e1c001b70 con 0x7f7e241430c0 2024-03-25T18:30:26.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.029+0000 7f7e1bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:26.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.029+0000 7f7e1bfff640 1 -- 172.21.15.99:0/2476472374 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f7e1c001f80 con 0x7f7e241430c0 2024-03-25T18:30:26.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.029+0000 7f7e2b2e4640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:26.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.029+0000 7f7e2b2e4640 4 set_mon_vals no callback set 2024-03-25T18:30:26.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.029+0000 7f7e1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:26.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.029+0000 7f7e1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:26.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.029+0000 7f7e1bfff640 10 monclient: dump: 2024-03-25T18:30:26.031 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:26.031 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:26.031 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:26.031 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:26.031 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:26.031 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:26.031 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:26.031 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:26.031 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:26.031 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:26.033 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.030+0000 7f7e2b2e4640 10 monclient: shutdown 2024-03-25T18:30:26.033 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.030+0000 7f7e2b2e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:26.033 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.030+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/2476472374 >> v1:172.21.15.99:6789/0 conn(0x7f7e241430c0 legacy=0x7f7e2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:26.033 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.030+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/2476472374 shutdown_connections 2024-03-25T18:30:26.033 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.030+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/2476472374 >> 172.21.15.99:0/2476472374 conn(0x7f7e240b7ea0 msgr2=0x7f7e240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:26.033 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.030+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/2476472374 shutdown_connections 2024-03-25T18:30:26.034 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.030+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/2476472374 wait complete. 2024-03-25T18:30:26.034 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.031+0000 7f7e2b2e4640 10 monclient: build_initial_monmap 2024-03-25T18:30:26.034 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.031+0000 7f7e2b2e4640 10 monclient: monmap: 2024-03-25T18:30:26.034 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:26.034 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:26.034 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:30:26.034 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:30:26.034 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:26.034 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:30:26.034 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:26.034 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:30:26.034 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:30:26.034 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:26.034 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.031+0000 7f7e2b2e4640 1 librados: starting msgr at 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.031+0000 7f7e2b2e4640 1 librados: starting objecter 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.031+0000 7f7e2b2e4640 1 Processor -- start 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.031+0000 7f7e2b2e4640 1 -- start start 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.031+0000 7f7e2b2e4640 1 librados: setting wanted keys 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.031+0000 7f7e2b2e4640 1 librados: calling monclient init 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.031+0000 7f7e2b2e4640 10 monclient: init 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.032+0000 7f7e2b2e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.032+0000 7f7e2b2e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.032+0000 7f7e2b2e4640 10 monclient(hunting): picked mon.noname-b con 0x7f7e240a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.032+0000 7f7e2b2e4640 10 monclient(hunting): picked mon.noname-a con 0x7f7e2413f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.032+0000 7f7e2b2e4640 10 monclient(hunting): picked mon.noname-c con 0x7f7e241430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.032+0000 7f7e2b2e4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e24137ef0 con 0x7f7e2413f710 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.032+0000 7f7e2b2e4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e24134720 con 0x7f7e240a98f0 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.032+0000 7f7e2b2e4640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e241348b0 con 0x7f7e241430c0 2024-03-25T18:30:26.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.032+0000 7f7e2b2e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:26.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.032+0000 7f7e2b2e4640 10 monclient(hunting): authenticate will time out at 7644.200264s 2024-03-25T18:30:26.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.035+0000 7f7e29080640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f7e240a98f0 0x7f7e241380a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:55114/0 (socket says 172.21.15.99:55114) 2024-03-25T18:30:26.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.036+0000 7f7e29080640 1 -- 172.21.15.99:0/3236935616 learned_addr learned my addr 172.21.15.99:0/3236935616 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:26.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.038+0000 7f7e19ffb640 10 client.?.objecter ms_handle_connect 0x7f7e240a98f0 2024-03-25T18:30:26.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.038+0000 7f7e19ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:26.046 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.045+0000 7f7e19ffb640 10 client.?.objecter ms_handle_connect 0x7f7e241430c0 2024-03-25T18:30:26.046 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.045+0000 7f7e19ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:26.046 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.045+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076836038 0 0) 0x7f7e24134720 con 0x7f7e240a98f0 2024-03-25T18:30:26.046 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.045+0000 7f7e19ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:26.046 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.045+0000 7f7e19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:26.046 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.045+0000 7f7e19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:26.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.045+0000 7f7e19ffb640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:30:26.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.045+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e08004eb0 con 0x7f7e240a98f0 2024-03-25T18:30:26.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.052+0000 7f7e19ffb640 10 client.?.objecter ms_handle_connect 0x7f7e2413f710 2024-03-25T18:30:26.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.052+0000 7f7e19ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:26.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.054+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 266961733 0 0) 0x7f7e08004eb0 con 0x7f7e240a98f0 2024-03-25T18:30:26.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.054+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e24134720 con 0x7f7e240a98f0 2024-03-25T18:30:26.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.054+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f7e20001a10 con 0x7f7e240a98f0 2024-03-25T18:30:26.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.054+0000 7f7e19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:26.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.055+0000 7f7e19ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:26.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.055+0000 7f7e19ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:26.056 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:26.056 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:26.056 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:26.056 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:26.056 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:26.057 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:26.057 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:26.057 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:26.057 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:26.057 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:26.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.055+0000 7f7e19ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:26.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.057+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733468432 0 0) 0x7f7e241348b0 con 0x7f7e241430c0 2024-03-25T18:30:26.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.057+0000 7f7e19ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:26.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.057+0000 7f7e19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:26.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.057+0000 7f7e19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:26.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.057+0000 7f7e19ffb640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:30:26.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.058+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e08004eb0 con 0x7f7e241430c0 2024-03-25T18:30:26.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3819531728 0 0) 0x7f7e24134720 con 0x7f7e240a98f0 2024-03-25T18:30:26.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 >> v1:172.21.15.99:6790/0 conn(0x7f7e241430c0 legacy=0x7f7e24134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:26.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 >> v1:172.21.15.99:6789/0 conn(0x7f7e2413f710 legacy=0x7f7e241377e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:26.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e19ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:30:26.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e19ffb640 1 monclient: found mon.b 2024-03-25T18:30:26.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:26.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e19ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:26.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e24134a40 con 0x7f7e240a98f0 2024-03-25T18:30:26.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e19ffb640 10 monclient: _finish_auth 0 2024-03-25T18:30:26.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e19ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:30:26.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:26.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e2b2e4640 5 monclient: authenticate success, global_id 4453 2024-03-25T18:30:26.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e2b2e4640 10 monclient: _renew_subs 2024-03-25T18:30:26.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e2b2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:26.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/3236935616 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e241daaa0 con 0x7f7e240a98f0 2024-03-25T18:30:26.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e2b2e4640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:26.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e2b2e4640 10 monclient: _renew_subs 2024-03-25T18:30:26.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e2b2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:26.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/3236935616 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e241db020 con 0x7f7e240a98f0 2024-03-25T18:30:26.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.069+0000 7f7e2b2e4640 1 librados: init done 2024-03-25T18:30:26.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.070+0000 7f7e2b2e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:26.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.070+0000 7f7e2b2e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:26.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.070+0000 7f7e2b2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:26.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.070+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/3236935616 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e241387b0 con 0x7f7e240a98f0 2024-03-25T18:30:26.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.349+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e20001c90 con 0x7f7e240a98f0 2024-03-25T18:30:26.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.349+0000 7f7e19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:26.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.349+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f7e20003a60 con 0x7f7e240a98f0 2024-03-25T18:30:26.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.349+0000 7f7e19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:26.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.349+0000 7f7e19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:26.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.349+0000 7f7e19ffb640 10 monclient: dump: 2024-03-25T18:30:26.351 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:26.351 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:26.351 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:26.351 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:26.351 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:26.352 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:26.352 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:26.352 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:26.352 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:26.352 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:26.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.349+0000 7f7e1b7fe640 4 set_mon_vals no callback set 2024-03-25T18:30:26.352 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.350+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f7e20003c40 con 0x7f7e240a98f0 2024-03-25T18:30:26.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.351+0000 7f7e2887f640 10 monclient: get_auth_request con 0x7f7e0806dd90 auth_method 0 2024-03-25T18:30:26.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.351+0000 7f7e19ffb640 10 client.4453.objecter ms_handle_connect 0x7f7e0806dd90 2024-03-25T18:30:26.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.355+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f7e200548e0 con 0x7f7e240a98f0 2024-03-25T18:30:26.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.355+0000 7f7e19ffb640 10 client.4453.objecter ms_dispatch 0x7f7e24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:26.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.355+0000 7f7e19ffb640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:26.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.355+0000 7f7e19ffb640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:26.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.355+0000 7f7e19ffb640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-25T18:30:26.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.356+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e20050220 con 0x7f7e240a98f0 2024-03-25T18:30:26.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.356+0000 7f7e19ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:26.357 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.356+0000 7f7e19ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:26.607 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.606+0000 7f7e2b2e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:26.608 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.606+0000 7f7e2b2e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:26.608 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.606+0000 7f7e2b2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:26.608 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.606+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/3236935616 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e24138bb0 con 0x7f7e240a98f0 2024-03-25T18:30:26.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.629+0000 7f7e19ffb640 1 -- 172.21.15.99:0/3236935616 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f7e20052320 con 0x7f7e240a98f0 2024-03-25T18:30:26.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.629+0000 7f7e19ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:26.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.629+0000 7f7e19ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:26.631 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:30:26.631 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:26.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.632+0000 7f7e2b2e4640 10 librados: watch_flush enter 2024-03-25T18:30:26.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.632+0000 7f7e2b2e4640 10 librados: watch_flush exit 2024-03-25T18:30:26.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.632+0000 7f7e2b2e4640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-25T18:30:26.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.632+0000 7f7e2b2e4640 10 client.4453.objecter successfully canceled tick 2024-03-25T18:30:26.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.632+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/3236935616 >> v1:172.21.15.190:6800/4220409699 conn(0x7f7e0806dd90 legacy=0x7f7e08070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:26.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.632+0000 7f7e2b2e4640 10 monclient: shutdown 2024-03-25T18:30:26.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.632+0000 7f7e2b2e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:26.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.632+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/3236935616 >> v1:172.21.15.190:6789/0 conn(0x7f7e240a98f0 legacy=0x7f7e241380a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:26.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.632+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/3236935616 shutdown_connections 2024-03-25T18:30:26.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.632+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/3236935616 >> 172.21.15.99:0/3236935616 conn(0x7f7e240b7ea0 msgr2=0x7f7e2414ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:26.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.632+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/3236935616 shutdown_connections 2024-03-25T18:30:26.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.632+0000 7f7e2b2e4640 1 -- 172.21.15.99:0/3236935616 wait complete. 2024-03-25T18:30:26.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:26.632+0000 7f7e2b2e4640 1 librados: shutdown 2024-03-25T18:30:26.644 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:29.226 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:05:29.818 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:29.232 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:05:29.824 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:29.278 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:05:29.871 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:29.297 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:05:29.856 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:29.306 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:05:29.865 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:29.314 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:05:29.906 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:29.328 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:05:29.920 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:29.334 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:05:29.893 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:29.360 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:05:29.952 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:33.451 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:33.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.570+0000 7f9f5e647640 10 monclient: get_monmap_and_config 2024-03-25T18:30:33.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.570+0000 7f9f5e647640 10 monclient: build_initial_monmap 2024-03-25T18:30:33.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.570+0000 7f9f5e647640 10 monclient: monmap: 2024-03-25T18:30:33.571 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:33.571 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:33.571 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:30:33.571178+0000 2024-03-25T18:30:33.571 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:30:33.571178+0000 2024-03-25T18:30:33.571 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:33.571 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:33.572 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:33.572 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:30:33.572 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:30:33.572 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:33.573 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.571+0000 7f9f5e647640 1 Processor -- start 2024-03-25T18:30:33.573 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.571+0000 7f9f5e647640 1 -- start start 2024-03-25T18:30:33.573 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.571+0000 7f9f5e647640 10 monclient: init 2024-03-25T18:30:33.573 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.572+0000 7f9f5e647640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:33.573 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.572+0000 7f9f5e647640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:33.573 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.572+0000 7f9f5e647640 10 monclient(hunting): picked mon.noname-a con 0x7f9f58061890 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:33.573 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.572+0000 7f9f5e647640 10 monclient(hunting): picked mon.noname-b con 0x7f9f5806e740 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:33.573 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.572+0000 7f9f5e647640 10 monclient(hunting): picked mon.noname-c con 0x7f9f580ae280 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:33.574 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.572+0000 7f9f5e647640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f580ae830 con 0x7f9f58061890 2024-03-25T18:30:33.574 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.572+0000 7f9f5e647640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5806f720 con 0x7f9f580ae280 2024-03-25T18:30:33.574 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.572+0000 7f9f5e647640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f580ada80 con 0x7f9f5806e740 2024-03-25T18:30:33.574 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.572+0000 7f9f5e647640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:33.574 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.572+0000 7f9f5e647640 10 monclient(hunting): authenticate will time out at 7651.740189s 2024-03-25T18:30:33.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.579+0000 7f9f57fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f9f58061890 0x7f9f5813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:36320/0 (socket says 172.21.15.99:36320) 2024-03-25T18:30:33.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.579+0000 7f9f57fff640 1 -- 172.21.15.99:0/2294578078 learned_addr learned my addr 172.21.15.99:0/2294578078 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:33.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.598+0000 7f9f56ffd640 1 -- 172.21.15.99:0/2294578078 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484038978 0 0) 0x7f9f5806f720 con 0x7f9f580ae280 2024-03-25T18:30:33.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.598+0000 7f9f56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:33.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.598+0000 7f9f56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:33.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.598+0000 7f9f56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:33.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.598+0000 7f9f56ffd640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:30:33.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.598+0000 7f9f56ffd640 1 -- 172.21.15.99:0/2294578078 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f40003520 con 0x7f9f580ae280 2024-03-25T18:30:33.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.600+0000 7f9f56ffd640 1 -- 172.21.15.99:0/2294578078 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707037227 0 0) 0x7f9f580ae830 con 0x7f9f58061890 2024-03-25T18:30:33.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.600+0000 7f9f56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:33.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.600+0000 7f9f56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:33.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.600+0000 7f9f56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:33.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.600+0000 7f9f56ffd640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:30:33.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.600+0000 7f9f56ffd640 1 -- 172.21.15.99:0/2294578078 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f5806f720 con 0x7f9f58061890 2024-03-25T18:30:33.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 1 -- 172.21.15.99:0/2294578078 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2516236019 0 0) 0x7f9f40003520 con 0x7f9f580ae280 2024-03-25T18:30:33.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 1 -- 172.21.15.99:0/2294578078 >> v1:172.21.15.99:6790/0 conn(0x7f9f5806e740 legacy=0x7f9f58145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:33.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 1 -- 172.21.15.99:0/2294578078 >> v1:172.21.15.99:6789/0 conn(0x7f9f58061890 legacy=0x7f9f5813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:33.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:33.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 1 monclient: found mon.noname-c 2024-03-25T18:30:33.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:33.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:30:33.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 1 -- 172.21.15.99:0/2294578078 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5813fbd0 con 0x7f9f580ae280 2024-03-25T18:30:33.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:33.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:33.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:33.611 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 1 -- 172.21.15.99:0/2294578078 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9f50001a10 con 0x7f9f580ae280 2024-03-25T18:30:33.611 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:33.611 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:33.611 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f56ffd640 10 monclient: dump: 2024-03-25T18:30:33.611 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:33.611 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:33.611 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:33.611 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:33.611 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:33.611 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:33.611 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:33.611 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:33.611 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:33.611 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:33.612 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f5e647640 5 monclient: authenticate success, global_id 4456 2024-03-25T18:30:33.612 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.608+0000 7f9f5e647640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:33.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.614+0000 7f9f56ffd640 1 -- 172.21.15.99:0/2294578078 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f50001b70 con 0x7f9f580ae280 2024-03-25T18:30:33.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.614+0000 7f9f56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:33.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.614+0000 7f9f56ffd640 1 -- 172.21.15.99:0/2294578078 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9f50001f80 con 0x7f9f580ae280 2024-03-25T18:30:33.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.614+0000 7f9f56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:33.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.614+0000 7f9f56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:33.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.614+0000 7f9f56ffd640 10 monclient: dump: 2024-03-25T18:30:33.620 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:33.620 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:33.620 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:33.620 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:33.620 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:33.620 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:33.620 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:33.620 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:33.620 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:33.620 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:33.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.614+0000 7f9f5e647640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:33.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.614+0000 7f9f5e647640 4 set_mon_vals no callback set 2024-03-25T18:30:33.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.615+0000 7f9f5e647640 10 monclient: shutdown 2024-03-25T18:30:33.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.615+0000 7f9f5e647640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:33.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.615+0000 7f9f5e647640 1 -- 172.21.15.99:0/2294578078 >> v1:172.21.15.190:6789/0 conn(0x7f9f580ae280 legacy=0x7f9f58145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:33.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.615+0000 7f9f5e647640 1 -- 172.21.15.99:0/2294578078 shutdown_connections 2024-03-25T18:30:33.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.615+0000 7f9f5e647640 1 -- 172.21.15.99:0/2294578078 >> 172.21.15.99:0/2294578078 conn(0x7f9f580b0ef0 msgr2=0x7f9f580b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:33.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.615+0000 7f9f5e647640 1 -- 172.21.15.99:0/2294578078 shutdown_connections 2024-03-25T18:30:33.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.615+0000 7f9f5e647640 1 -- 172.21.15.99:0/2294578078 wait complete. 2024-03-25T18:30:33.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.616+0000 7f9f5e647640 10 monclient: build_initial_monmap 2024-03-25T18:30:33.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.616+0000 7f9f5e647640 10 monclient: monmap: 2024-03-25T18:30:33.621 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:33.621 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:33.621 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:30:33.621 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:30:33.622 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:33.622 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:30:33.622 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:33.622 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:30:33.622 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:30:33.622 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:33.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.616+0000 7f9f5e647640 1 librados: starting msgr at 2024-03-25T18:30:33.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.616+0000 7f9f5e647640 1 librados: starting objecter 2024-03-25T18:30:33.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.616+0000 7f9f5e647640 1 Processor -- start 2024-03-25T18:30:33.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.617+0000 7f9f5e647640 1 -- start start 2024-03-25T18:30:33.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.617+0000 7f9f5e647640 1 librados: setting wanted keys 2024-03-25T18:30:33.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.617+0000 7f9f5e647640 1 librados: calling monclient init 2024-03-25T18:30:33.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.617+0000 7f9f5e647640 10 monclient: init 2024-03-25T18:30:33.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.617+0000 7f9f5e647640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:33.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.617+0000 7f9f5e647640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:33.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.617+0000 7f9f5e647640 10 monclient(hunting): picked mon.noname-c con 0x7f9f58061890 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:33.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.617+0000 7f9f5e647640 10 monclient(hunting): picked mon.noname-a con 0x7f9f580ae280 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:33.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.617+0000 7f9f5e647640 10 monclient(hunting): picked mon.noname-b con 0x7f9f58145b20 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:33.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.617+0000 7f9f5e647640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f58140ab0 con 0x7f9f580ae280 2024-03-25T18:30:33.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.617+0000 7f9f5e647640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f581430b0 con 0x7f9f58145b20 2024-03-25T18:30:33.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.617+0000 7f9f5e647640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f58141240 con 0x7f9f58061890 2024-03-25T18:30:33.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.617+0000 7f9f5e647640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:33.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.617+0000 7f9f5e647640 10 monclient(hunting): authenticate will time out at 7651.785818s 2024-03-25T18:30:33.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.632+0000 7f9f577fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f9f580ae280 0x7f9f58141f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:36366/0 (socket says 172.21.15.99:36366) 2024-03-25T18:30:33.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.632+0000 7f9f577fe640 1 -- 172.21.15.99:0/394801171 learned_addr learned my addr 172.21.15.99:0/394801171 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:33.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.633+0000 7f9f54ff9640 10 client.?.objecter ms_handle_connect 0x7f9f580ae280 2024-03-25T18:30:33.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.633+0000 7f9f54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:33.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.641+0000 7f9f54ff9640 10 client.?.objecter ms_handle_connect 0x7f9f58145b20 2024-03-25T18:30:33.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.641+0000 7f9f54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:33.652 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.651+0000 7f9f54ff9640 10 client.?.objecter ms_handle_connect 0x7f9f58061890 2024-03-25T18:30:33.652 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.651+0000 7f9f54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:33.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.655+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182323900 0 0) 0x7f9f58140ab0 con 0x7f9f580ae280 2024-03-25T18:30:33.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.655+0000 7f9f54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:33.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.655+0000 7f9f54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:33.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.655+0000 7f9f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:33.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.655+0000 7f9f54ff9640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:30:33.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.655+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f44004eb0 con 0x7f9f580ae280 2024-03-25T18:30:33.658 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.657+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199173411 0 0) 0x7f9f581430b0 con 0x7f9f58145b20 2024-03-25T18:30:33.658 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.657+0000 7f9f54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:33.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.657+0000 7f9f54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:33.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.657+0000 7f9f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:33.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.657+0000 7f9f54ff9640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:30:33.659 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.657+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f58140ab0 con 0x7f9f58145b20 2024-03-25T18:30:33.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.663+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051460006 0 0) 0x7f9f58141240 con 0x7f9f58061890 2024-03-25T18:30:33.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.663+0000 7f9f54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:33.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.663+0000 7f9f54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:33.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.663+0000 7f9f54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:33.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.663+0000 7f9f54ff9640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:30:33.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.663+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f581430b0 con 0x7f9f58061890 2024-03-25T18:30:33.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.664+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1521931888 0 0) 0x7f9f44004eb0 con 0x7f9f580ae280 2024-03-25T18:30:33.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.664+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f58141240 con 0x7f9f580ae280 2024-03-25T18:30:33.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.664+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9f4c001a30 con 0x7f9f580ae280 2024-03-25T18:30:33.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.664+0000 7f9f54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:33.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.664+0000 7f9f54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:33.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.664+0000 7f9f54ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:33.666 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:33.666 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:33.666 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:33.666 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:33.666 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:33.666 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:33.666 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:33.666 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:33.666 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:33.667 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:33.667 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.664+0000 7f9f54ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:33.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.671+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1642438762 0 0) 0x7f9f58140ab0 con 0x7f9f58145b20 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.671+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f44004eb0 con 0x7f9f58145b20 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.671+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9f50001e80 con 0x7f9f58145b20 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.671+0000 7f9f54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.671+0000 7f9f54ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.671+0000 7f9f54ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:33.673 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:33.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3412171086 0 0) 0x7f9f58141240 con 0x7f9f580ae280 2024-03-25T18:30:33.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 >> v1:172.21.15.99:6790/0 conn(0x7f9f58061890 legacy=0x7f9f58143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:33.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 >> v1:172.21.15.190:6789/0 conn(0x7f9f58145b20 legacy=0x7f9f58142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:33.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f54ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:33.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f54ff9640 1 monclient: found mon.a 2024-03-25T18:30:33.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:33.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f54ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:33.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f58141520 con 0x7f9f580ae280 2024-03-25T18:30:33.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f54ff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:33.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f54ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:33.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:33.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f5e647640 5 monclient: authenticate success, global_id 4443 2024-03-25T18:30:33.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f5e647640 10 monclient: _renew_subs 2024-03-25T18:30:33.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f5e647640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:33.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.673+0000 7f9f5e647640 1 -- 172.21.15.99:0/394801171 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f58141800 con 0x7f9f580ae280 2024-03-25T18:30:33.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.674+0000 7f9f5e647640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:33.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.674+0000 7f9f5e647640 10 monclient: _renew_subs 2024-03-25T18:30:33.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.674+0000 7f9f5e647640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:33.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.674+0000 7f9f5e647640 1 -- 172.21.15.99:0/394801171 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f58142d20 con 0x7f9f580ae280 2024-03-25T18:30:33.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.674+0000 7f9f5e647640 1 librados: init done 2024-03-25T18:30:33.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.674+0000 7f9f5e647640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:33.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.674+0000 7f9f5e647640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:33.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.674+0000 7f9f5e647640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:33.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.674+0000 7f9f5e647640 1 -- 172.21.15.99:0/394801171 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5806c520 con 0x7f9f580ae280 2024-03-25T18:30:33.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.966+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f4c001c70 con 0x7f9f580ae280 2024-03-25T18:30:33.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.966+0000 7f9f54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:33.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.966+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9f4c003a60 con 0x7f9f580ae280 2024-03-25T18:30:33.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.966+0000 7f9f54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:33.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.966+0000 7f9f54ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:33.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.966+0000 7f9f54ff9640 10 monclient: dump: 2024-03-25T18:30:33.968 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:33.968 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:33.968 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:33.968 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:33.968 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:33.968 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:33.968 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:33.968 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:33.968 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:33.968 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:33.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.966+0000 7f9f567fc640 4 set_mon_vals no callback set 2024-03-25T18:30:33.969 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.967+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f9f4c003c80 con 0x7f9f580ae280 2024-03-25T18:30:33.969 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.968+0000 7f9f57fff640 10 monclient: get_auth_request con 0x7f9f4406df70 auth_method 0 2024-03-25T18:30:33.969 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.968+0000 7f9f54ff9640 10 client.4443.objecter ms_handle_connect 0x7f9f4406df70 2024-03-25T18:30:33.975 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.974+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f9f4c0548a0 con 0x7f9f580ae280 2024-03-25T18:30:33.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.974+0000 7f9f54ff9640 10 client.4443.objecter ms_dispatch 0x7f9f58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:33.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.974+0000 7f9f54ff9640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:33.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.974+0000 7f9f54ff9640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:33.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.974+0000 7f9f54ff9640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-25T18:30:33.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.974+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f4c0503b0 con 0x7f9f580ae280 2024-03-25T18:30:33.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.974+0000 7f9f54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:33.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:33.974+0000 7f9f54ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:34.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.217+0000 7f9f5e647640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:34.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.217+0000 7f9f5e647640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:34.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.217+0000 7f9f5e647640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:34.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.217+0000 7f9f5e647640 1 -- 172.21.15.99:0/394801171 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f580b20d0 con 0x7f9f580ae280 2024-03-25T18:30:34.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.248+0000 7f9f54ff9640 1 -- 172.21.15.99:0/394801171 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f9f4c0522e0 con 0x7f9f580ae280 2024-03-25T18:30:34.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.248+0000 7f9f54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:34.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.248+0000 7f9f54ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:34.250 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:30:34.250 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:34.252 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.251+0000 7f9f5e647640 10 librados: watch_flush enter 2024-03-25T18:30:34.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.251+0000 7f9f5e647640 10 librados: watch_flush exit 2024-03-25T18:30:34.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.251+0000 7f9f5e647640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-25T18:30:34.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.251+0000 7f9f5e647640 10 client.4443.objecter successfully canceled tick 2024-03-25T18:30:34.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.251+0000 7f9f5e647640 1 -- 172.21.15.99:0/394801171 >> v1:172.21.15.190:6800/4220409699 conn(0x7f9f4406df70 legacy=0x7f9f44070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:34.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.251+0000 7f9f5e647640 10 monclient: shutdown 2024-03-25T18:30:34.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.251+0000 7f9f5e647640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:34.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.251+0000 7f9f5e647640 1 -- 172.21.15.99:0/394801171 >> v1:172.21.15.99:6789/0 conn(0x7f9f580ae280 legacy=0x7f9f58141f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:34.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.251+0000 7f9f5e647640 1 -- 172.21.15.99:0/394801171 shutdown_connections 2024-03-25T18:30:34.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.251+0000 7f9f5e647640 1 -- 172.21.15.99:0/394801171 >> 172.21.15.99:0/394801171 conn(0x7f9f580b0ef0 msgr2=0x7f9f5806fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:34.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.251+0000 7f9f5e647640 1 -- 172.21.15.99:0/394801171 shutdown_connections 2024-03-25T18:30:34.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.251+0000 7f9f5e647640 1 -- 172.21.15.99:0/394801171 wait complete. 2024-03-25T18:30:34.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:34.251+0000 7f9f5e647640 1 librados: shutdown 2024-03-25T18:30:34.265 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:41.072 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:41.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.185+0000 7fd524a35640 10 monclient: get_monmap_and_config 2024-03-25T18:30:41.187 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.185+0000 7fd524a35640 10 monclient: build_initial_monmap 2024-03-25T18:30:41.187 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.185+0000 7fd524a35640 10 monclient: monmap: 2024-03-25T18:30:41.187 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:41.187 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:41.187 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:30:41.186675+0000 2024-03-25T18:30:41.187 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:30:41.186675+0000 2024-03-25T18:30:41.187 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:41.187 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:41.187 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:41.187 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:30:41.187 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:30:41.187 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:41.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.186+0000 7fd524a35640 1 Processor -- start 2024-03-25T18:30:41.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.186+0000 7fd524a35640 1 -- start start 2024-03-25T18:30:41.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.186+0000 7fd524a35640 10 monclient: init 2024-03-25T18:30:41.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.187+0000 7fd524a35640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:41.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.187+0000 7fd524a35640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:41.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.187+0000 7fd524a35640 10 monclient(hunting): picked mon.noname-b con 0x7fd5200b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:41.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.187+0000 7fd524a35640 10 monclient(hunting): picked mon.noname-a con 0x7fd5200b1730 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:41.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.187+0000 7fd524a35640 10 monclient(hunting): picked mon.noname-c con 0x7fd52014be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:41.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.187+0000 7fd524a35640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5200af940 con 0x7fd5200b1730 2024-03-25T18:30:41.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.187+0000 7fd524a35640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5200b0a70 con 0x7fd52014be90 2024-03-25T18:30:41.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.187+0000 7fd524a35640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5200b0d50 con 0x7fd5200b4d60 2024-03-25T18:30:41.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.187+0000 7fd524a35640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:41.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.187+0000 7fd524a35640 10 monclient(hunting): authenticate will time out at 7659.355297s 2024-03-25T18:30:41.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.191+0000 7fd51e59c640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7fd5200b4d60 0x7fd52013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:38712/0 (socket says 172.21.15.99:38712) 2024-03-25T18:30:41.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.191+0000 7fd51e59c640 1 -- 172.21.15.99:0/1139961843 learned_addr learned my addr 172.21.15.99:0/1139961843 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:41.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.202+0000 7fd51dd9b640 1 -- 172.21.15.99:0/1139961843 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116240041 0 0) 0x7fd5200b0d50 con 0x7fd5200b4d60 2024-03-25T18:30:41.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.202+0000 7fd51dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:30:41.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.202+0000 7fd51dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:41.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.202+0000 7fd51dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:41.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.202+0000 7fd51dd9b640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:30:41.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.202+0000 7fd51dd9b640 1 -- 172.21.15.99:0/1139961843 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd50c003520 con 0x7fd5200b4d60 2024-03-25T18:30:41.212 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 1 -- 172.21.15.99:0/1139961843 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2212603157 0 0) 0x7fd50c003520 con 0x7fd5200b4d60 2024-03-25T18:30:41.212 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 1 -- 172.21.15.99:0/1139961843 >> v1:172.21.15.190:6789/0 conn(0x7fd52014be90 legacy=0x7fd52014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:41.212 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 1 -- 172.21.15.99:0/1139961843 >> v1:172.21.15.99:6789/0 conn(0x7fd5200b1730 legacy=0x7fd5200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 1 monclient: found mon.noname-b 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 1 -- 172.21.15.99:0/1139961843 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5200b1030 con 0x7fd5200b4d60 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 1 -- 172.21.15.99:0/1139961843 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fd510001690 con 0x7fd5200b4d60 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd51dd9b640 10 monclient: dump: 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:41.213 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd524a35640 5 monclient: authenticate success, global_id 4457 2024-03-25T18:30:41.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.211+0000 7fd524a35640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:41.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.223+0000 7fd51dd9b640 1 -- 172.21.15.99:0/1139961843 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5100017f0 con 0x7fd5200b4d60 2024-03-25T18:30:41.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.223+0000 7fd51dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:41.225 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.223+0000 7fd51dd9b640 1 -- 172.21.15.99:0/1139961843 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fd510001c00 con 0x7fd5200b4d60 2024-03-25T18:30:41.225 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.223+0000 7fd524a35640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:41.225 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.223+0000 7fd524a35640 4 set_mon_vals no callback set 2024-03-25T18:30:41.225 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.223+0000 7fd51dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:41.225 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.223+0000 7fd51dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:41.225 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.223+0000 7fd51dd9b640 10 monclient: dump: 2024-03-25T18:30:41.225 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:41.225 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:41.225 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:41.225 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:41.225 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:41.225 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:41.225 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:41.225 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:41.226 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:41.226 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:41.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.224+0000 7fd524a35640 10 monclient: shutdown 2024-03-25T18:30:41.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.224+0000 7fd524a35640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:41.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.224+0000 7fd524a35640 1 -- 172.21.15.99:0/1139961843 >> v1:172.21.15.99:6790/0 conn(0x7fd5200b4d60 legacy=0x7fd52013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:41.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.224+0000 7fd524a35640 1 -- 172.21.15.99:0/1139961843 shutdown_connections 2024-03-25T18:30:41.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.224+0000 7fd524a35640 1 -- 172.21.15.99:0/1139961843 >> 172.21.15.99:0/1139961843 conn(0x7fd5200617a0 msgr2=0x7fd520061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:41.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.224+0000 7fd524a35640 1 -- 172.21.15.99:0/1139961843 shutdown_connections 2024-03-25T18:30:41.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.224+0000 7fd524a35640 1 -- 172.21.15.99:0/1139961843 wait complete. 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.225+0000 7fd524a35640 10 monclient: build_initial_monmap 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.225+0000 7fd524a35640 10 monclient: monmap: 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 1 librados: starting msgr at 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 1 librados: starting objecter 2024-03-25T18:30:41.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 1 Processor -- start 2024-03-25T18:30:41.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 1 -- start start 2024-03-25T18:30:41.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 1 librados: setting wanted keys 2024-03-25T18:30:41.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 1 librados: calling monclient init 2024-03-25T18:30:41.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 10 monclient: init 2024-03-25T18:30:41.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:41.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:41.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 10 monclient(hunting): picked mon.noname-a con 0x7fd5200b1730 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:41.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 10 monclient(hunting): picked mon.noname-b con 0x7fd5200b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:41.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 10 monclient(hunting): picked mon.noname-c con 0x7fd52014be90 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:41.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5201361e0 con 0x7fd5200b1730 2024-03-25T18:30:41.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd520145080 con 0x7fd5200b4d60 2024-03-25T18:30:41.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd520145210 con 0x7fd52014be90 2024-03-25T18:30:41.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:41.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.226+0000 7fd524a35640 10 monclient(hunting): authenticate will time out at 7659.394839s 2024-03-25T18:30:41.237 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.236+0000 7fd517fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fd5200b4d60 0x7fd5201d76c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:34838/0 (socket says 172.21.15.99:34838) 2024-03-25T18:30:41.237 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.236+0000 7fd517fff640 1 -- 172.21.15.99:0/4049300569 learned_addr learned my addr 172.21.15.99:0/4049300569 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:41.241 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.240+0000 7fd516ffd640 10 client.?.objecter ms_handle_connect 0x7fd5200b4d60 2024-03-25T18:30:41.241 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.240+0000 7fd516ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:41.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.242+0000 7fd516ffd640 10 client.?.objecter ms_handle_connect 0x7fd5200b1730 2024-03-25T18:30:41.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.242+0000 7fd516ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:41.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.242+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438823214 0 0) 0x7fd520145080 con 0x7fd5200b4d60 2024-03-25T18:30:41.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.242+0000 7fd516ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:41.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.242+0000 7fd516ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:41.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.242+0000 7fd516ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:41.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.242+0000 7fd516ffd640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:30:41.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.242+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd500004eb0 con 0x7fd5200b4d60 2024-03-25T18:30:41.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.243+0000 7fd516ffd640 10 client.?.objecter ms_handle_connect 0x7fd52014be90 2024-03-25T18:30:41.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.243+0000 7fd516ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:41.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.253+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4169657610 0 0) 0x7fd500004eb0 con 0x7fd5200b4d60 2024-03-25T18:30:41.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.253+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd520145080 con 0x7fd5200b4d60 2024-03-25T18:30:41.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.253+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fd508001a30 con 0x7fd5200b4d60 2024-03-25T18:30:41.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.253+0000 7fd516ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:41.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.253+0000 7fd516ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:41.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.253+0000 7fd516ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:41.255 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:41.255 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:41.255 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:41.255 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:41.255 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:41.255 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:41.256 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:41.256 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:41.256 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:41.256 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:41.256 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.253+0000 7fd516ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:41.256 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.256+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228323919 0 0) 0x7fd5201361e0 con 0x7fd5200b1730 2024-03-25T18:30:41.257 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.256+0000 7fd516ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:41.257 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.256+0000 7fd516ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:41.257 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.256+0000 7fd516ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:41.257 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.256+0000 7fd516ffd640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:30:41.257 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.256+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd500004eb0 con 0x7fd5200b1730 2024-03-25T18:30:41.266 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.264+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853742616 0 0) 0x7fd520145210 con 0x7fd52014be90 2024-03-25T18:30:41.266 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.265+0000 7fd516ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:41.266 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.265+0000 7fd516ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:41.266 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.265+0000 7fd516ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:41.266 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.265+0000 7fd516ffd640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:30:41.266 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.265+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5201361e0 con 0x7fd52014be90 2024-03-25T18:30:41.271 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.269+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1231240727 0 0) 0x7fd520145080 con 0x7fd5200b4d60 2024-03-25T18:30:41.271 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.269+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 >> v1:172.21.15.99:6790/0 conn(0x7fd52014be90 legacy=0x7fd5201e2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:41.271 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.269+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 >> v1:172.21.15.99:6789/0 conn(0x7fd5200b1730 legacy=0x7fd52012ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:41.271 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.269+0000 7fd516ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:41.271 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.269+0000 7fd516ffd640 1 monclient: found mon.b 2024-03-25T18:30:41.271 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd516ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:41.271 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd516ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:41.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5201306a0 con 0x7fd5200b4d60 2024-03-25T18:30:41.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd516ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:41.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd516ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:41.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd516ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:41.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd524a35640 5 monclient: authenticate success, global_id 4465 2024-03-25T18:30:41.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd524a35640 10 monclient: _renew_subs 2024-03-25T18:30:41.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd524a35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:41.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd524a35640 1 -- 172.21.15.99:0/4049300569 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd520145440 con 0x7fd5200b4d60 2024-03-25T18:30:41.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd524a35640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:41.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd524a35640 10 monclient: _renew_subs 2024-03-25T18:30:41.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd524a35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:41.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd524a35640 1 -- 172.21.15.99:0/4049300569 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd520137ea0 con 0x7fd5200b4d60 2024-03-25T18:30:41.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd524a35640 1 librados: init done 2024-03-25T18:30:41.273 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.270+0000 7fd524a35640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:41.273 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.271+0000 7fd524a35640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:41.273 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.271+0000 7fd524a35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:41.273 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.271+0000 7fd524a35640 1 -- 172.21.15.99:0/4049300569 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd520130890 con 0x7fd5200b4d60 2024-03-25T18:30:41.546 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.545+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd508001cb0 con 0x7fd5200b4d60 2024-03-25T18:30:41.546 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.545+0000 7fd516ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:41.546 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.545+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fd508003a60 con 0x7fd5200b4d60 2024-03-25T18:30:41.546 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.545+0000 7fd516ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:41.547 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.545+0000 7fd516ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:41.547 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.545+0000 7fd516ffd640 10 monclient: dump: 2024-03-25T18:30:41.547 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:41.547 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:41.547 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:41.547 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:41.547 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:41.547 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:41.547 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:41.547 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:41.547 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:41.547 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:41.547 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.545+0000 7fd51d59a640 4 set_mon_vals no callback set 2024-03-25T18:30:41.548 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.546+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fd508003c40 con 0x7fd5200b4d60 2024-03-25T18:30:41.548 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.547+0000 7fd51e59c640 10 monclient: get_auth_request con 0x7fd50006de80 auth_method 0 2024-03-25T18:30:41.548 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.547+0000 7fd516ffd640 10 client.4465.objecter ms_handle_connect 0x7fd50006de80 2024-03-25T18:30:41.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.552+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fd508054870 con 0x7fd5200b4d60 2024-03-25T18:30:41.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.552+0000 7fd516ffd640 10 client.4465.objecter ms_dispatch 0x7fd520000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:41.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.552+0000 7fd516ffd640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:41.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.552+0000 7fd516ffd640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:41.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.552+0000 7fd516ffd640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-25T18:30:41.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.552+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd508050380 con 0x7fd5200b4d60 2024-03-25T18:30:41.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.552+0000 7fd516ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:41.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.552+0000 7fd516ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:41.804 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.803+0000 7fd524a35640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:41.804 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.803+0000 7fd524a35640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:41.804 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.803+0000 7fd524a35640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:30:41.804 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.803+0000 7fd524a35640 1 -- 172.21.15.99:0/4049300569 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5200627b0 con 0x7fd5200b4d60 2024-03-25T18:30:41.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.823+0000 7fd516ffd640 1 -- 172.21.15.99:0/4049300569 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fd5080522b0 con 0x7fd5200b4d60 2024-03-25T18:30:41.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.823+0000 7fd516ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:41.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.823+0000 7fd516ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:41.825 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:30:41.825 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:41.827 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.825+0000 7fd524a35640 10 librados: watch_flush enter 2024-03-25T18:30:41.827 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.825+0000 7fd524a35640 10 librados: watch_flush exit 2024-03-25T18:30:41.827 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.825+0000 7fd524a35640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-25T18:30:41.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.825+0000 7fd524a35640 10 client.4465.objecter successfully canceled tick 2024-03-25T18:30:41.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.825+0000 7fd524a35640 1 -- 172.21.15.99:0/4049300569 >> v1:172.21.15.190:6800/4220409699 conn(0x7fd50006de80 legacy=0x7fd500070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:41.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.825+0000 7fd524a35640 10 monclient: shutdown 2024-03-25T18:30:41.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.825+0000 7fd524a35640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:41.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.825+0000 7fd524a35640 1 -- 172.21.15.99:0/4049300569 >> v1:172.21.15.190:6789/0 conn(0x7fd5200b4d60 legacy=0x7fd5201d76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:41.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.825+0000 7fd524a35640 1 -- 172.21.15.99:0/4049300569 shutdown_connections 2024-03-25T18:30:41.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.825+0000 7fd524a35640 1 -- 172.21.15.99:0/4049300569 >> 172.21.15.99:0/4049300569 conn(0x7fd5200617a0 msgr2=0x7fd5200b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:41.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.825+0000 7fd524a35640 1 -- 172.21.15.99:0/4049300569 shutdown_connections 2024-03-25T18:30:41.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.825+0000 7fd524a35640 1 -- 172.21.15.99:0/4049300569 wait complete. 2024-03-25T18:30:41.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:41.826+0000 7fd524a35640 1 librados: shutdown 2024-03-25T18:30:41.838 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:48.644 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:48.756 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.755+0000 7f69fa3b5640 10 monclient: get_monmap_and_config 2024-03-25T18:30:48.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.755+0000 7f69fa3b5640 10 monclient: build_initial_monmap 2024-03-25T18:30:48.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.755+0000 7f69fa3b5640 10 monclient: monmap: 2024-03-25T18:30:48.757 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:48.757 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:48.757 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:30:48.756777+0000 2024-03-25T18:30:48.757 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:30:48.756777+0000 2024-03-25T18:30:48.757 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:48.757 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:48.757 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:48.757 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:30:48.757 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:30:48.757 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:48.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.756+0000 7f69fa3b5640 1 Processor -- start 2024-03-25T18:30:48.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.756+0000 7f69fa3b5640 1 -- start start 2024-03-25T18:30:48.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.756+0000 7f69fa3b5640 10 monclient: init 2024-03-25T18:30:48.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.757+0000 7f69fa3b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:48.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.757+0000 7f69fa3b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:48.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.757+0000 7f69fa3b5640 10 monclient(hunting): picked mon.noname-c con 0x7f69f40705b0 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:48.760 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.757+0000 7f69fa3b5640 10 monclient(hunting): picked mon.noname-a con 0x7f69f41402d0 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:48.760 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.757+0000 7f69fa3b5640 10 monclient(hunting): picked mon.noname-b con 0x7f69f4149ae0 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:48.760 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.757+0000 7f69fa3b5640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f40718c0 con 0x7f69f41402d0 2024-03-25T18:30:48.760 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.757+0000 7f69fa3b5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f40714c0 con 0x7f69f40705b0 2024-03-25T18:30:48.760 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.757+0000 7f69fa3b5640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f4072b50 con 0x7f69f4149ae0 2024-03-25T18:30:48.760 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.757+0000 7f69fa3b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:48.760 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.757+0000 7f69fa3b5640 10 monclient(hunting): authenticate will time out at 7666.925126s 2024-03-25T18:30:48.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.766+0000 7f69f8952640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f69f4149ae0 0x7f69f414bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:37406/0 (socket says 172.21.15.99:37406) 2024-03-25T18:30:48.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.766+0000 7f69f8952640 1 -- 172.21.15.99:0/3304570481 learned_addr learned my addr 172.21.15.99:0/3304570481 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:48.777 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.776+0000 7f69f2ffd640 1 -- 172.21.15.99:0/3304570481 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974841708 0 0) 0x7f69f4072b50 con 0x7f69f4149ae0 2024-03-25T18:30:48.777 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.776+0000 7f69f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:48.777 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.776+0000 7f69f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:48.777 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.776+0000 7f69f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:48.777 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.776+0000 7f69f2ffd640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:30:48.777 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.776+0000 7f69f2ffd640 1 -- 172.21.15.99:0/3304570481 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69dc003520 con 0x7f69f4149ae0 2024-03-25T18:30:48.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.777+0000 7f69f2ffd640 1 -- 172.21.15.99:0/3304570481 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947330847 0 0) 0x7f69f40714c0 con 0x7f69f40705b0 2024-03-25T18:30:48.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.777+0000 7f69f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:48.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.777+0000 7f69f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:48.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.777+0000 7f69f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:48.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.777+0000 7f69f2ffd640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:30:48.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.777+0000 7f69f2ffd640 1 -- 172.21.15.99:0/3304570481 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f4072b50 con 0x7f69f40705b0 2024-03-25T18:30:48.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.781+0000 7f69f2ffd640 1 -- 172.21.15.99:0/3304570481 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3809801139 0 0) 0x7f69dc003520 con 0x7f69f4149ae0 2024-03-25T18:30:48.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.781+0000 7f69f2ffd640 1 -- 172.21.15.99:0/3304570481 >> v1:172.21.15.190:6789/0 conn(0x7f69f40705b0 legacy=0x7f69f413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:48.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.781+0000 7f69f2ffd640 1 -- 172.21.15.99:0/3304570481 >> v1:172.21.15.99:6789/0 conn(0x7f69f41402d0 legacy=0x7f69f4140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:48.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.781+0000 7f69f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:48.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.781+0000 7f69f2ffd640 1 monclient: found mon.noname-b 2024-03-25T18:30:48.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.781+0000 7f69f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:48.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.781+0000 7f69f2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:30:48.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.781+0000 7f69f2ffd640 1 -- 172.21.15.99:0/3304570481 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f413dd10 con 0x7f69f4149ae0 2024-03-25T18:30:48.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.781+0000 7f69f2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:48.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.781+0000 7f69f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:48.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.781+0000 7f69f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:48.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.781+0000 7f69fa3b5640 5 monclient: authenticate success, global_id 4463 2024-03-25T18:30:48.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.781+0000 7f69fa3b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:48.796 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.795+0000 7f69f2ffd640 1 -- 172.21.15.99:0/3304570481 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f69ec0019d0 con 0x7f69f4149ae0 2024-03-25T18:30:48.796 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.795+0000 7f69f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:48.796 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.795+0000 7f69f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:48.796 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.795+0000 7f69f2ffd640 10 monclient: dump: 2024-03-25T18:30:48.796 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:48.796 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:48.797 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:48.797 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:48.797 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:48.797 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:48.797 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:48.797 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:48.797 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:48.797 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:48.797 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.795+0000 7f69f2ffd640 1 -- 172.21.15.99:0/3304570481 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69ec001c20 con 0x7f69f4149ae0 2024-03-25T18:30:48.797 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.795+0000 7f69fa3b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:48.797 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.795+0000 7f69f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:48.797 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.795+0000 7f69fa3b5640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:48.797 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.795+0000 7f69f2ffd640 1 -- 172.21.15.99:0/3304570481 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f69ec001fb0 con 0x7f69f4149ae0 2024-03-25T18:30:48.798 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.795+0000 7f69f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:48.798 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.795+0000 7f69fa3b5640 4 set_mon_vals no callback set 2024-03-25T18:30:48.798 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.795+0000 7f69f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:48.798 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.795+0000 7f69f2ffd640 10 monclient: dump: 2024-03-25T18:30:48.798 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:48.798 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:48.798 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:48.798 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:48.798 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:48.798 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:48.798 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:48.799 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:48.799 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:48.799 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:48.801 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.796+0000 7f69fa3b5640 10 monclient: shutdown 2024-03-25T18:30:48.801 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.796+0000 7f69fa3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:48.801 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.796+0000 7f69fa3b5640 1 -- 172.21.15.99:0/3304570481 >> v1:172.21.15.99:6790/0 conn(0x7f69f4149ae0 legacy=0x7f69f414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:48.801 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.796+0000 7f69fa3b5640 1 -- 172.21.15.99:0/3304570481 shutdown_connections 2024-03-25T18:30:48.801 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.796+0000 7f69fa3b5640 1 -- 172.21.15.99:0/3304570481 >> 172.21.15.99:0/3304570481 conn(0x7f69f40b5d10 msgr2=0x7f69f40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:48.801 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.796+0000 7f69fa3b5640 1 -- 172.21.15.99:0/3304570481 shutdown_connections 2024-03-25T18:30:48.801 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.796+0000 7f69fa3b5640 1 -- 172.21.15.99:0/3304570481 wait complete. 2024-03-25T18:30:48.801 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.797+0000 7f69fa3b5640 10 monclient: build_initial_monmap 2024-03-25T18:30:48.801 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.797+0000 7f69fa3b5640 10 monclient: monmap: 2024-03-25T18:30:48.801 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:48.801 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:48.801 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:30:48.802 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:30:48.802 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:48.802 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:30:48.802 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:48.802 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:30:48.802 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:30:48.802 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:48.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.797+0000 7f69fa3b5640 1 librados: starting msgr at 2024-03-25T18:30:48.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.797+0000 7f69fa3b5640 1 librados: starting objecter 2024-03-25T18:30:48.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.797+0000 7f69fa3b5640 1 Processor -- start 2024-03-25T18:30:48.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.797+0000 7f69fa3b5640 1 -- start start 2024-03-25T18:30:48.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.797+0000 7f69fa3b5640 1 librados: setting wanted keys 2024-03-25T18:30:48.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.797+0000 7f69fa3b5640 1 librados: calling monclient init 2024-03-25T18:30:48.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.797+0000 7f69fa3b5640 10 monclient: init 2024-03-25T18:30:48.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.798+0000 7f69fa3b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:48.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.798+0000 7f69fa3b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:48.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.798+0000 7f69fa3b5640 10 monclient(hunting): picked mon.noname-b con 0x7f69f40705b0 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:48.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.798+0000 7f69fa3b5640 10 monclient(hunting): picked mon.noname-c con 0x7f69f41402d0 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:48.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.798+0000 7f69fa3b5640 10 monclient(hunting): picked mon.noname-a con 0x7f69f4149ae0 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:48.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.798+0000 7f69fa3b5640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f40af4e0 con 0x7f69f4149ae0 2024-03-25T18:30:48.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.798+0000 7f69fa3b5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f406d870 con 0x7f69f40705b0 2024-03-25T18:30:48.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.798+0000 7f69fa3b5640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f406c950 con 0x7f69f41402d0 2024-03-25T18:30:48.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.798+0000 7f69fa3b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:48.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.798+0000 7f69fa3b5640 10 monclient(hunting): authenticate will time out at 7666.966560s 2024-03-25T18:30:48.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.807+0000 7f69f37fe640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f69f41402d0 0x7f69f41d7cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:37442/0 (socket says 172.21.15.99:37442) 2024-03-25T18:30:48.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.807+0000 7f69f37fe640 1 -- 172.21.15.99:0/1717063098 learned_addr learned my addr 172.21.15.99:0/1717063098 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:48.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.808+0000 7f69f0ff9640 10 client.?.objecter ms_handle_connect 0x7f69f41402d0 2024-03-25T18:30:48.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.808+0000 7f69f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:48.815 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.814+0000 7f69f0ff9640 10 client.?.objecter ms_handle_connect 0x7f69f4149ae0 2024-03-25T18:30:48.815 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.814+0000 7f69f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:48.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.816+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276932411 0 0) 0x7f69f406c950 con 0x7f69f41402d0 2024-03-25T18:30:48.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.816+0000 7f69f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:48.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.816+0000 7f69f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:48.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.816+0000 7f69f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:48.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.816+0000 7f69f0ff9640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:30:48.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.816+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e0004eb0 con 0x7f69f41402d0 2024-03-25T18:30:48.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.817+0000 7f69f0ff9640 10 client.?.objecter ms_handle_connect 0x7f69f40705b0 2024-03-25T18:30:48.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.817+0000 7f69f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:48.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.822+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342948934 0 0) 0x7f69f40af4e0 con 0x7f69f4149ae0 2024-03-25T18:30:48.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.822+0000 7f69f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:48.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.822+0000 7f69f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:48.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.822+0000 7f69f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:48.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.822+0000 7f69f0ff9640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:30:48.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.822+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f406c950 con 0x7f69f4149ae0 2024-03-25T18:30:48.831 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.830+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820516612 0 0) 0x7f69f406d870 con 0x7f69f40705b0 2024-03-25T18:30:48.831 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.830+0000 7f69f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:48.831 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.830+0000 7f69f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:48.831 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.830+0000 7f69f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:48.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.830+0000 7f69f0ff9640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:30:48.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.830+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f40af4e0 con 0x7f69f40705b0 2024-03-25T18:30:48.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.834+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2557566097 0 0) 0x7f69f406c950 con 0x7f69f4149ae0 2024-03-25T18:30:48.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.834+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69f406d870 con 0x7f69f4149ae0 2024-03-25T18:30:48.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.834+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f69ec001f30 con 0x7f69f4149ae0 2024-03-25T18:30:48.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.834+0000 7f69f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:48.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.834+0000 7f69f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:48.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.834+0000 7f69f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:48.835 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:48.835 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:48.836 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:48.836 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:48.836 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:48.836 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:48.836 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:48.836 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:48.836 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:48.836 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:48.836 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.834+0000 7f69f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:48.838 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.837+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3408135402 0 0) 0x7f69e0004eb0 con 0x7f69f41402d0 2024-03-25T18:30:48.838 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.837+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69f406c950 con 0x7f69f41402d0 2024-03-25T18:30:48.838 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.837+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f69e8001a30 con 0x7f69f41402d0 2024-03-25T18:30:48.838 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.837+0000 7f69f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:48.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.837+0000 7f69f0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:48.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.837+0000 7f69f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:48.839 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:48.839 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:48.839 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:48.839 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:48.839 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:48.839 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:48.839 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:48.839 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:48.839 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:48.839 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:48.852 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3325026824 0 0) 0x7f69f406d870 con 0x7f69f4149ae0 2024-03-25T18:30:48.852 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 >> v1:172.21.15.99:6790/0 conn(0x7f69f41402d0 legacy=0x7f69f41d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 >> v1:172.21.15.190:6789/0 conn(0x7f69f40705b0 legacy=0x7f69f41d75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69f0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69f0ff9640 1 monclient: found mon.a 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69f0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f406cae0 con 0x7f69f4149ae0 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69f0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69f0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69fa3b5640 5 monclient: authenticate success, global_id 4455 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69fa3b5640 10 monclient: _renew_subs 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69fa3b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69fa3b5640 1 -- 172.21.15.99:0/1717063098 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f406beb0 con 0x7f69f4149ae0 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69fa3b5640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:48.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69fa3b5640 10 monclient: _renew_subs 2024-03-25T18:30:48.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69fa3b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:48.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69fa3b5640 1 -- 172.21.15.99:0/1717063098 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f413f450 con 0x7f69f4149ae0 2024-03-25T18:30:48.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.851+0000 7f69fa3b5640 1 librados: init done 2024-03-25T18:30:48.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.852+0000 7f69fa3b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:48.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.852+0000 7f69fa3b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:48.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.852+0000 7f69fa3b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:48.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:48.852+0000 7f69fa3b5640 1 -- 172.21.15.99:0/1717063098 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f40af2b0 con 0x7f69f4149ae0 2024-03-25T18:30:49.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.108+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69ec001580 con 0x7f69f4149ae0 2024-03-25T18:30:49.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.108+0000 7f69f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:49.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.108+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f69ec002090 con 0x7f69f4149ae0 2024-03-25T18:30:49.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.108+0000 7f69f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:49.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.108+0000 7f69f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:49.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.108+0000 7f69f0ff9640 10 monclient: dump: 2024-03-25T18:30:49.110 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:49.110 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:49.110 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:49.110 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:49.110 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:49.110 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:49.110 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:49.110 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:49.110 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:49.111 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:49.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.108+0000 7f69f27fc640 4 set_mon_vals no callback set 2024-03-25T18:30:49.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.109+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f69ec01acc0 con 0x7f69f4149ae0 2024-03-25T18:30:49.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.109+0000 7f69f3fff640 10 monclient: get_auth_request con 0x7f69e006df60 auth_method 0 2024-03-25T18:30:49.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.110+0000 7f69f0ff9640 10 client.4455.objecter ms_handle_connect 0x7f69e006df60 2024-03-25T18:30:49.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.113+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f69ec054890 con 0x7f69f4149ae0 2024-03-25T18:30:49.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.113+0000 7f69f0ff9640 10 client.4455.objecter ms_dispatch 0x7f69f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:49.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.113+0000 7f69f0ff9640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:49.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.113+0000 7f69f0ff9640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:49.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.113+0000 7f69f0ff9640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-25T18:30:49.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.113+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69ec0501d0 con 0x7f69f4149ae0 2024-03-25T18:30:49.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.113+0000 7f69f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:49.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.113+0000 7f69f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:49.314 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:05:49.907 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:49.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.330+0000 7f69fa3b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:49.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.330+0000 7f69fa3b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:49.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.330+0000 7f69fa3b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:49.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.330+0000 7f69fa3b5640 1 -- 172.21.15.99:0/1717063098 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69f413f7a0 con 0x7f69f4149ae0 2024-03-25T18:30:49.365 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:05:49.957 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:49.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.388+0000 7f69f0ff9640 1 -- 172.21.15.99:0/1717063098 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f69ec0522d0 con 0x7f69f4149ae0 2024-03-25T18:30:49.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.388+0000 7f69f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:49.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.388+0000 7f69f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:49.390 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:30:49.391 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:49.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.390+0000 7f69fa3b5640 10 librados: watch_flush enter 2024-03-25T18:30:49.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.390+0000 7f69fa3b5640 10 librados: watch_flush exit 2024-03-25T18:30:49.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.390+0000 7f69fa3b5640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-25T18:30:49.394 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.390+0000 7f69fa3b5640 10 client.4455.objecter successfully canceled tick 2024-03-25T18:30:49.394 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.391+0000 7f69fa3b5640 1 -- 172.21.15.99:0/1717063098 >> v1:172.21.15.190:6800/4220409699 conn(0x7f69e006df60 legacy=0x7f69e0070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:49.394 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.391+0000 7f69fa3b5640 10 monclient: shutdown 2024-03-25T18:30:49.394 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.391+0000 7f69fa3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:49.394 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.391+0000 7f69fa3b5640 1 -- 172.21.15.99:0/1717063098 >> v1:172.21.15.99:6789/0 conn(0x7f69f4149ae0 legacy=0x7f69f41e1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:49.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.391+0000 7f69fa3b5640 1 -- 172.21.15.99:0/1717063098 shutdown_connections 2024-03-25T18:30:49.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.391+0000 7f69fa3b5640 1 -- 172.21.15.99:0/1717063098 >> 172.21.15.99:0/1717063098 conn(0x7f69f40b5d10 msgr2=0x7f69f40b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:49.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.391+0000 7f69fa3b5640 1 -- 172.21.15.99:0/1717063098 shutdown_connections 2024-03-25T18:30:49.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.391+0000 7f69fa3b5640 1 -- 172.21.15.99:0/1717063098 wait complete. 2024-03-25T18:30:49.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:49.391+0000 7f69fa3b5640 1 librados: shutdown 2024-03-25T18:30:49.403 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:49.411 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:05:50.003 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:49.423 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:05:50.015 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:49.429 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:05:50.022 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:49.445 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:05:50.004 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:49.450 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:05:50.009 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:49.450 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:05:50.043 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:49.474 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:05:50.033 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:56.210 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:56.324 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.323+0000 7fedf248b640 10 monclient: get_monmap_and_config 2024-03-25T18:30:56.325 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.323+0000 7fedf248b640 10 monclient: build_initial_monmap 2024-03-25T18:30:56.325 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.323+0000 7fedf248b640 10 monclient: monmap: 2024-03-25T18:30:56.325 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:56.325 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:56.325 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:30:56.324612+0000 2024-03-25T18:30:56.325 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:30:56.324612+0000 2024-03-25T18:30:56.325 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:56.325 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:56.325 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:56.325 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:30:56.325 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:30:56.326 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:56.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.324+0000 7fedf248b640 1 Processor -- start 2024-03-25T18:30:56.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.324+0000 7fedf248b640 1 -- start start 2024-03-25T18:30:56.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.324+0000 7fedf248b640 10 monclient: init 2024-03-25T18:30:56.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.325+0000 7fedf248b640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:56.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.325+0000 7fedf248b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:56.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.325+0000 7fedf248b640 10 monclient(hunting): picked mon.noname-a con 0x7fedec0ae6e0 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:56.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.325+0000 7fedf248b640 10 monclient(hunting): picked mon.noname-b con 0x7fedec06f080 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:56.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.325+0000 7fedf248b640 10 monclient(hunting): picked mon.noname-c con 0x7fedec0704e0 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:56.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.325+0000 7fedf248b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedec0718d0 con 0x7fedec0ae6e0 2024-03-25T18:30:56.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.325+0000 7fedf248b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedec06fa30 con 0x7fedec0704e0 2024-03-25T18:30:56.328 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.325+0000 7fedf248b640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedec070090 con 0x7fedec06f080 2024-03-25T18:30:56.328 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.325+0000 7fedf248b640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:56.328 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.325+0000 7fedf248b640 10 monclient(hunting): authenticate will time out at 7674.493521s 2024-03-25T18:30:56.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.336+0000 7fedebfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fedec0ae6e0 0x7fedec13c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:39700/0 (socket says 172.21.15.99:39700) 2024-03-25T18:30:56.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.336+0000 7fedebfff640 1 -- 172.21.15.99:0/2417625653 learned_addr learned my addr 172.21.15.99:0/2417625653 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:56.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.350+0000 7fedeaffd640 1 -- 172.21.15.99:0/2417625653 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406486199 0 0) 0x7fedec0718d0 con 0x7fedec0ae6e0 2024-03-25T18:30:56.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.350+0000 7fedeaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:56.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.350+0000 7fedeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:56.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.350+0000 7fedeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:56.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.350+0000 7fedeaffd640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:30:56.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.350+0000 7fedeaffd640 1 -- 172.21.15.99:0/2417625653 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd4003520 con 0x7fedec0ae6e0 2024-03-25T18:30:56.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.357+0000 7fedeaffd640 1 -- 172.21.15.99:0/2417625653 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771315308 0 0) 0x7fedec06fa30 con 0x7fedec0704e0 2024-03-25T18:30:56.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.357+0000 7fedeaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:56.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.357+0000 7fedeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:56.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.357+0000 7fedeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:56.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.357+0000 7fedeaffd640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:30:56.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.357+0000 7fedeaffd640 1 -- 172.21.15.99:0/2417625653 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedec0718d0 con 0x7fedec0704e0 2024-03-25T18:30:56.361 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 1 -- 172.21.15.99:0/2417625653 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4267297497 0 0) 0x7fedd4003520 con 0x7fedec0ae6e0 2024-03-25T18:30:56.361 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 1 -- 172.21.15.99:0/2417625653 >> v1:172.21.15.99:6790/0 conn(0x7fedec06f080 legacy=0x7fedec071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:56.361 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 1 -- 172.21.15.99:0/2417625653 >> v1:172.21.15.190:6789/0 conn(0x7fedec0704e0 legacy=0x7fedec14a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:56.361 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:56.361 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 1 monclient: found mon.noname-a 2024-03-25T18:30:56.361 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:56.361 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:30:56.361 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 1 -- 172.21.15.99:0/2417625653 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec06f510 con 0x7fedec0ae6e0 2024-03-25T18:30:56.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:56.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:56.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:56.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 1 -- 172.21.15.99:0/2417625653 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7feddc001690 con 0x7fedec0ae6e0 2024-03-25T18:30:56.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedf248b640 5 monclient: authenticate success, global_id 4458 2024-03-25T18:30:56.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedf248b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:56.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:56.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:56.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedeaffd640 10 monclient: dump: 2024-03-25T18:30:56.362 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:56.362 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:56.362 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:56.362 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:56.363 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:56.363 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:56.363 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:56.363 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:56.363 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:56.363 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:56.363 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.359+0000 7fedf248b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:56.366 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.365+0000 7fedeaffd640 1 -- 172.21.15.99:0/2417625653 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feddc0017f0 con 0x7fedec0ae6e0 2024-03-25T18:30:56.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.365+0000 7fedeaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:56.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.365+0000 7fedeaffd640 1 -- 172.21.15.99:0/2417625653 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7feddc001c00 con 0x7fedec0ae6e0 2024-03-25T18:30:56.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.365+0000 7fedeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:56.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.365+0000 7fedeaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:56.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.365+0000 7fedeaffd640 10 monclient: dump: 2024-03-25T18:30:56.367 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:56.367 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:56.367 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:56.367 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:56.367 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:56.367 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:56.367 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:56.367 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:56.367 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:56.368 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:56.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.365+0000 7fedf248b640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:56.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.365+0000 7fedf248b640 4 set_mon_vals no callback set 2024-03-25T18:30:56.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.366+0000 7fedf248b640 10 monclient: shutdown 2024-03-25T18:30:56.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.366+0000 7fedf248b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:56.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.366+0000 7fedf248b640 1 -- 172.21.15.99:0/2417625653 >> v1:172.21.15.99:6789/0 conn(0x7fedec0ae6e0 legacy=0x7fedec13c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:56.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.366+0000 7fedf248b640 1 -- 172.21.15.99:0/2417625653 shutdown_connections 2024-03-25T18:30:56.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.366+0000 7fedf248b640 1 -- 172.21.15.99:0/2417625653 >> 172.21.15.99:0/2417625653 conn(0x7fedec06b630 msgr2=0x7fedec0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:56.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.366+0000 7fedf248b640 1 -- 172.21.15.99:0/2417625653 shutdown_connections 2024-03-25T18:30:56.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.366+0000 7fedf248b640 1 -- 172.21.15.99:0/2417625653 wait complete. 2024-03-25T18:30:56.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.367+0000 7fedf248b640 10 monclient: build_initial_monmap 2024-03-25T18:30:56.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.367+0000 7fedf248b640 10 monclient: monmap: 2024-03-25T18:30:56.369 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:30:56.370 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:56.370 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:30:56.370 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:30:56.370 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:56.370 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:30:56.370 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:30:56.370 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:30:56.370 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:30:56.370 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:56.370 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.368+0000 7fedf248b640 1 librados: starting msgr at 2024-03-25T18:30:56.370 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.368+0000 7fedf248b640 1 librados: starting objecter 2024-03-25T18:30:56.370 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.368+0000 7fedf248b640 1 Processor -- start 2024-03-25T18:30:56.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.368+0000 7fedf248b640 1 -- start start 2024-03-25T18:30:56.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.368+0000 7fedf248b640 1 librados: setting wanted keys 2024-03-25T18:30:56.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.368+0000 7fedf248b640 1 librados: calling monclient init 2024-03-25T18:30:56.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.368+0000 7fedf248b640 10 monclient: init 2024-03-25T18:30:56.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.369+0000 7fedf248b640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:56.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.369+0000 7fedf248b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:56.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.369+0000 7fedf248b640 10 monclient(hunting): picked mon.noname-a con 0x7fedec0704e0 addr v1:172.21.15.99:6789/0 2024-03-25T18:30:56.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.369+0000 7fedf248b640 10 monclient(hunting): picked mon.noname-c con 0x7fedec0ae6e0 addr v1:172.21.15.99:6790/0 2024-03-25T18:30:56.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.369+0000 7fedf248b640 10 monclient(hunting): picked mon.noname-b con 0x7fedec0aef70 addr v1:172.21.15.190:6789/0 2024-03-25T18:30:56.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.369+0000 7fedf248b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedec142300 con 0x7fedec0704e0 2024-03-25T18:30:56.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.369+0000 7fedf248b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedec142f10 con 0x7fedec0aef70 2024-03-25T18:30:56.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.369+0000 7fedf248b640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedec0a9f20 con 0x7fedec0ae6e0 2024-03-25T18:30:56.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.369+0000 7fedf248b640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:56.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.369+0000 7fedf248b640 10 monclient(hunting): authenticate will time out at 7674.537590s 2024-03-25T18:30:56.381 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.380+0000 7fedebfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fedec0704e0 0x7fedec141a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:39752/0 (socket says 172.21.15.99:39752) 2024-03-25T18:30:56.381 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.380+0000 7fedebfff640 1 -- 172.21.15.99:0/2465231278 learned_addr learned my addr 172.21.15.99:0/2465231278 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:30:56.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.382+0000 7fede8ff9640 10 client.?.objecter ms_handle_connect 0x7fedec0704e0 2024-03-25T18:30:56.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.383+0000 7fede8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:56.385 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.384+0000 7fede8ff9640 10 client.?.objecter ms_handle_connect 0x7fedec0aef70 2024-03-25T18:30:56.385 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.384+0000 7fede8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:56.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.387+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121193107 0 0) 0x7fedec142300 con 0x7fedec0704e0 2024-03-25T18:30:56.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.387+0000 7fede8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:56.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.387+0000 7fede8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:56.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.387+0000 7fede8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:56.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.387+0000 7fede8ff9640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:30:56.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.387+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd8004eb0 con 0x7fedec0704e0 2024-03-25T18:30:56.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.389+0000 7fede8ff9640 10 client.?.objecter ms_handle_connect 0x7fedec0ae6e0 2024-03-25T18:30:56.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.389+0000 7fede8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:56.396 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.394+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185518769 0 0) 0x7fedec142f10 con 0x7fedec0aef70 2024-03-25T18:30:56.396 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.395+0000 7fede8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:56.396 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.395+0000 7fede8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:56.396 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.395+0000 7fede8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:56.396 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.395+0000 7fede8ff9640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:30:56.396 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.395+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedec142300 con 0x7fedec0aef70 2024-03-25T18:30:56.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.399+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220331440 0 0) 0x7fedec0a9f20 con 0x7fedec0ae6e0 2024-03-25T18:30:56.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.399+0000 7fede8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:56.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.399+0000 7fede8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:56.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.399+0000 7fede8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:56.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.399+0000 7fede8ff9640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:30:56.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.399+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedec142f10 con 0x7fedec0ae6e0 2024-03-25T18:30:56.415 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.414+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 111536741 0 0) 0x7fedd8004eb0 con 0x7fedec0704e0 2024-03-25T18:30:56.415 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.414+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedec0a9f20 con 0x7fedec0704e0 2024-03-25T18:30:56.415 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.414+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7feddc001bc0 con 0x7fedec0704e0 2024-03-25T18:30:56.415 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.414+0000 7fede8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:56.415 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.414+0000 7fede8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:56.415 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.414+0000 7fede8ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:56.415 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:56.416 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:56.416 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:56.416 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:56.416 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:56.416 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:56.416 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:56.416 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:56.416 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:56.416 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:56.416 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.414+0000 7fede8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:56.420 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.418+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2889457399 0 0) 0x7fedec142300 con 0x7fedec0aef70 2024-03-25T18:30:56.420 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.418+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedd8004eb0 con 0x7fedec0aef70 2024-03-25T18:30:56.420 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.419+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fede4001cb0 con 0x7fedec0aef70 2024-03-25T18:30:56.420 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.419+0000 7fede8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:56.420 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.419+0000 7fede8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:56.420 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.419+0000 7fede8ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:56.420 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:56.421 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:56.421 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:56.421 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:56.421 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:56.421 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:56.421 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:56.421 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:56.421 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:56.421 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:56.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.423+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1835756706 0 0) 0x7fedec142f10 con 0x7fedec0ae6e0 2024-03-25T18:30:56.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.423+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedec142300 con 0x7fedec0ae6e0 2024-03-25T18:30:56.425 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.423+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fede0001830 con 0x7fedec0ae6e0 2024-03-25T18:30:56.425 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.423+0000 7fede8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:56.425 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.423+0000 7fede8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:56.425 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.423+0000 7fede8ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:56.425 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:56.425 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:56.425 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:56.425 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:56.425 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:56.425 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:56.425 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:56.425 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:56.425 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:56.426 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:56.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2066390691 0 0) 0x7fedec0a9f20 con 0x7fedec0704e0 2024-03-25T18:30:56.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 >> v1:172.21.15.99:6790/0 conn(0x7fedec0ae6e0 legacy=0x7fedec142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:56.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 >> v1:172.21.15.190:6789/0 conn(0x7fedec0aef70 legacy=0x7fedec135330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:56.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fede8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:56.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fede8ff9640 1 monclient: found mon.a 2024-03-25T18:30:56.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fede8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:56.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fede8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:56.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec1365d0 con 0x7fedec0704e0 2024-03-25T18:30:56.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fede8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:56.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fede8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:56.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fede8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:56.430 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fedf248b640 5 monclient: authenticate success, global_id 4461 2024-03-25T18:30:56.430 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fedf248b640 10 monclient: _renew_subs 2024-03-25T18:30:56.430 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fedf248b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:56.430 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fedf248b640 1 -- 172.21.15.99:0/2465231278 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedec1d75f0 con 0x7fedec0704e0 2024-03-25T18:30:56.430 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fedf248b640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:56.430 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fedf248b640 10 monclient: _renew_subs 2024-03-25T18:30:56.430 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fedf248b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:56.430 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fedf248b640 1 -- 172.21.15.99:0/2465231278 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fedec135a40 con 0x7fedec0704e0 2024-03-25T18:30:56.430 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.427+0000 7fedf248b640 1 librados: init done 2024-03-25T18:30:56.431 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.428+0000 7fedf248b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:56.431 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.428+0000 7fedf248b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:56.431 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.428+0000 7fedf248b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:56.431 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.428+0000 7fedf248b640 1 -- 172.21.15.99:0/2465231278 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedec13cc70 con 0x7fedec0704e0 2024-03-25T18:30:56.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.744+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feddc0011f0 con 0x7fedec0704e0 2024-03-25T18:30:56.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.744+0000 7fede8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:56.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.744+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7feddc004c30 con 0x7fedec0704e0 2024-03-25T18:30:56.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.744+0000 7fede8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:56.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.744+0000 7fede8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:56.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.744+0000 7fede8ff9640 10 monclient: dump: 2024-03-25T18:30:56.746 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:30:56.746 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:30:56.746 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:56.746 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:30:56.746 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:56.746 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:30:56.746 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:30:56.746 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:30:56.746 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:30:56.746 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:30:56.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.744+0000 7fedea7fc640 4 set_mon_vals no callback set 2024-03-25T18:30:56.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.745+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7feddc01ac40 con 0x7fedec0704e0 2024-03-25T18:30:56.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.746+0000 7fedeb7fe640 10 monclient: get_auth_request con 0x7fedd806e080 auth_method 0 2024-03-25T18:30:56.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.746+0000 7fede8ff9640 10 client.4461.objecter ms_handle_connect 0x7fedd806e080 2024-03-25T18:30:56.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.751+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7feddc054920 con 0x7fedec0704e0 2024-03-25T18:30:56.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.752+0000 7fede8ff9640 10 client.4461.objecter ms_dispatch 0x7fedec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:56.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.752+0000 7fede8ff9640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:56.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.752+0000 7fede8ff9640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:56.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.752+0000 7fede8ff9640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-25T18:30:56.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.752+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feddc050260 con 0x7fedec0704e0 2024-03-25T18:30:56.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.752+0000 7fede8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:56.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:56.752+0000 7fede8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:57.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.003+0000 7fedf248b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:57.004 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.003+0000 7fedf248b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:57.005 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.003+0000 7fedf248b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:30:57.005 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.003+0000 7fedf248b640 1 -- 172.21.15.99:0/2465231278 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedec0627b0 con 0x7fedec0704e0 2024-03-25T18:30:57.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.076+0000 7fede8ff9640 1 -- 172.21.15.99:0/2465231278 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7feddc052360 con 0x7fedec0704e0 2024-03-25T18:30:57.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.076+0000 7fede8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:57.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.076+0000 7fede8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:57.078 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:30:57.078 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:57.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.079+0000 7fedf248b640 10 librados: watch_flush enter 2024-03-25T18:30:57.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.079+0000 7fedf248b640 10 librados: watch_flush exit 2024-03-25T18:30:57.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.079+0000 7fedf248b640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-25T18:30:57.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.079+0000 7fedf248b640 10 client.4461.objecter successfully canceled tick 2024-03-25T18:30:57.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.079+0000 7fedf248b640 1 -- 172.21.15.99:0/2465231278 >> v1:172.21.15.190:6800/4220409699 conn(0x7fedd806e080 legacy=0x7fedd8070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:57.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.079+0000 7fedf248b640 10 monclient: shutdown 2024-03-25T18:30:57.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.079+0000 7fedf248b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:57.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.079+0000 7fedf248b640 1 -- 172.21.15.99:0/2465231278 >> v1:172.21.15.99:6789/0 conn(0x7fedec0704e0 legacy=0x7fedec141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:57.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.079+0000 7fedf248b640 1 -- 172.21.15.99:0/2465231278 shutdown_connections 2024-03-25T18:30:57.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.079+0000 7fedf248b640 1 -- 172.21.15.99:0/2465231278 >> 172.21.15.99:0/2465231278 conn(0x7fedec06b630 msgr2=0x7fedec13c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:57.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.079+0000 7fedf248b640 1 -- 172.21.15.99:0/2465231278 shutdown_connections 2024-03-25T18:30:57.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.079+0000 7fedf248b640 1 -- 172.21.15.99:0/2465231278 wait complete. 2024-03-25T18:30:57.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:30:57.079+0000 7fedf248b640 1 librados: shutdown 2024-03-25T18:30:57.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:03.898 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:04.009 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.007+0000 7fb8554d6640 10 monclient: get_monmap_and_config 2024-03-25T18:31:04.009 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.007+0000 7fb8554d6640 10 monclient: build_initial_monmap 2024-03-25T18:31:04.009 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.007+0000 7fb8554d6640 10 monclient: monmap: 2024-03-25T18:31:04.009 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:04.009 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:04.009 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:31:04.008946+0000 2024-03-25T18:31:04.009 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:31:04.008946+0000 2024-03-25T18:31:04.009 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:04.009 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:04.010 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:04.010 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:31:04.010 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:31:04.010 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:04.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.008+0000 7fb8554d6640 1 Processor -- start 2024-03-25T18:31:04.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.008+0000 7fb8554d6640 1 -- start start 2024-03-25T18:31:04.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.008+0000 7fb8554d6640 10 monclient: init 2024-03-25T18:31:04.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.009+0000 7fb8554d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:04.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.009+0000 7fb8554d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:04.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.009+0000 7fb8554d6640 10 monclient(hunting): picked mon.noname-b con 0x7fb8500a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:04.011 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.009+0000 7fb8554d6640 10 monclient(hunting): picked mon.noname-a con 0x7fb85013f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:04.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.009+0000 7fb8554d6640 10 monclient(hunting): picked mon.noname-c con 0x7fb8501430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:04.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.009+0000 7fb8554d6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb850142520 con 0x7fb85013f710 2024-03-25T18:31:04.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.009+0000 7fb8554d6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8500b28a0 con 0x7fb8501430c0 2024-03-25T18:31:04.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.009+0000 7fb8554d6640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8500b2a30 con 0x7fb8500a98f0 2024-03-25T18:31:04.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.009+0000 7fb8554d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:04.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.009+0000 7fb8554d6640 10 monclient(hunting): authenticate will time out at 7682.177491s 2024-03-25T18:31:04.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.021+0000 7fb84f7fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb8501430c0 0x7fb85014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:55416/0 (socket says 172.21.15.99:55416) 2024-03-25T18:31:04.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.021+0000 7fb84f7fe640 1 -- 172.21.15.99:0/1666154627 learned_addr learned my addr 172.21.15.99:0/1666154627 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:04.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.040+0000 7fb84dffb640 1 -- 172.21.15.99:0/1666154627 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361371119 0 0) 0x7fb8500b28a0 con 0x7fb8501430c0 2024-03-25T18:31:04.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.040+0000 7fb84dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:04.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.040+0000 7fb84dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:04.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.040+0000 7fb84dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:04.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.040+0000 7fb84dffb640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:31:04.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.040+0000 7fb84dffb640 1 -- 172.21.15.99:0/1666154627 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb838003520 con 0x7fb8501430c0 2024-03-25T18:31:04.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.041+0000 7fb84dffb640 1 -- 172.21.15.99:0/1666154627 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655919334 0 0) 0x7fb850142520 con 0x7fb85013f710 2024-03-25T18:31:04.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.041+0000 7fb84dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:04.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.041+0000 7fb84dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:04.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.041+0000 7fb84dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:04.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.041+0000 7fb84dffb640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:31:04.043 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.041+0000 7fb84dffb640 1 -- 172.21.15.99:0/1666154627 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8500b28a0 con 0x7fb85013f710 2024-03-25T18:31:04.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 1 -- 172.21.15.99:0/1666154627 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 587192197 0 0) 0x7fb8500b28a0 con 0x7fb85013f710 2024-03-25T18:31:04.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 1 -- 172.21.15.99:0/1666154627 >> v1:172.21.15.99:6790/0 conn(0x7fb8500a98f0 legacy=0x7fb8500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 1 -- 172.21.15.99:0/1666154627 >> v1:172.21.15.190:6789/0 conn(0x7fb8501430c0 legacy=0x7fb85014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:04.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 1 monclient: found mon.noname-a 2024-03-25T18:31:04.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:04.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:31:04.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 1 -- 172.21.15.99:0/1666154627 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8500b44c0 con 0x7fb85013f710 2024-03-25T18:31:04.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:04.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:04.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:04.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 1 -- 172.21.15.99:0/1666154627 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb844001670 con 0x7fb85013f710 2024-03-25T18:31:04.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:04.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:04.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb84dffb640 10 monclient: dump: 2024-03-25T18:31:04.059 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:04.059 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:04.059 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:04.059 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:04.059 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:04.059 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:04.060 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:04.060 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:04.060 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:04.060 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:04.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb8554d6640 5 monclient: authenticate success, global_id 4464 2024-03-25T18:31:04.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.056+0000 7fb8554d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:04.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.078+0000 7fb84dffb640 1 -- 172.21.15.99:0/1666154627 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8440017d0 con 0x7fb85013f710 2024-03-25T18:31:04.080 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.078+0000 7fb84dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:04.080 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.078+0000 7fb84dffb640 1 -- 172.21.15.99:0/1666154627 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb844001ba0 con 0x7fb85013f710 2024-03-25T18:31:04.080 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.078+0000 7fb84dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:04.080 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.078+0000 7fb84dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:04.080 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.078+0000 7fb84dffb640 10 monclient: dump: 2024-03-25T18:31:04.081 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:04.081 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:04.081 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:04.081 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:04.082 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:04.082 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:04.082 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:04.082 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:04.082 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:04.082 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:04.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.078+0000 7fb8554d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:04.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.078+0000 7fb8554d6640 4 set_mon_vals no callback set 2024-03-25T18:31:04.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.079+0000 7fb8554d6640 10 monclient: shutdown 2024-03-25T18:31:04.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.079+0000 7fb8554d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:04.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.079+0000 7fb8554d6640 1 -- 172.21.15.99:0/1666154627 >> v1:172.21.15.99:6789/0 conn(0x7fb85013f710 legacy=0x7fb8501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.079+0000 7fb8554d6640 1 -- 172.21.15.99:0/1666154627 shutdown_connections 2024-03-25T18:31:04.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.079+0000 7fb8554d6640 1 -- 172.21.15.99:0/1666154627 >> 172.21.15.99:0/1666154627 conn(0x7fb8500b7ea0 msgr2=0x7fb8500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:04.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.079+0000 7fb8554d6640 1 -- 172.21.15.99:0/1666154627 shutdown_connections 2024-03-25T18:31:04.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.079+0000 7fb8554d6640 1 -- 172.21.15.99:0/1666154627 wait complete. 2024-03-25T18:31:04.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.080+0000 7fb8554d6640 10 monclient: build_initial_monmap 2024-03-25T18:31:04.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.080+0000 7fb8554d6640 10 monclient: monmap: 2024-03-25T18:31:04.086 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:04.086 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:04.086 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:31:04.086 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:31:04.086 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:04.086 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:31:04.086 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:04.086 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:31:04.086 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:31:04.086 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:04.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.080+0000 7fb8554d6640 1 librados: starting msgr at 2024-03-25T18:31:04.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.080+0000 7fb8554d6640 1 librados: starting objecter 2024-03-25T18:31:04.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.081+0000 7fb8554d6640 1 Processor -- start 2024-03-25T18:31:04.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.081+0000 7fb8554d6640 1 -- start start 2024-03-25T18:31:04.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.081+0000 7fb8554d6640 1 librados: setting wanted keys 2024-03-25T18:31:04.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.081+0000 7fb8554d6640 1 librados: calling monclient init 2024-03-25T18:31:04.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.081+0000 7fb8554d6640 10 monclient: init 2024-03-25T18:31:04.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.081+0000 7fb8554d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:04.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.081+0000 7fb8554d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:04.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.081+0000 7fb8554d6640 10 monclient(hunting): picked mon.noname-b con 0x7fb8500a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:04.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.081+0000 7fb8554d6640 10 monclient(hunting): picked mon.noname-c con 0x7fb85013f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:04.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.082+0000 7fb8554d6640 10 monclient(hunting): picked mon.noname-a con 0x7fb8501430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:04.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.082+0000 7fb8554d6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8501d7830 con 0x7fb8501430c0 2024-03-25T18:31:04.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.082+0000 7fb8554d6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8501d7500 con 0x7fb8500a98f0 2024-03-25T18:31:04.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.082+0000 7fb8554d6640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb850137970 con 0x7fb85013f710 2024-03-25T18:31:04.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.082+0000 7fb8554d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:04.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.082+0000 7fb8554d6640 10 monclient(hunting): authenticate will time out at 7682.249931s 2024-03-25T18:31:04.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.093+0000 7fb84effd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb8500a98f0 0x7fb850138710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:55438/0 (socket says 172.21.15.99:55438) 2024-03-25T18:31:04.094 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.093+0000 7fb84effd640 1 -- 172.21.15.99:0/3007810447 learned_addr learned my addr 172.21.15.99:0/3007810447 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:04.095 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.093+0000 7fb8377fe640 10 client.?.objecter ms_handle_connect 0x7fb8500a98f0 2024-03-25T18:31:04.095 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.093+0000 7fb8377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:04.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.098+0000 7fb8377fe640 10 client.?.objecter ms_handle_connect 0x7fb8501430c0 2024-03-25T18:31:04.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.098+0000 7fb8377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:04.102 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.101+0000 7fb8377fe640 10 client.?.objecter ms_handle_connect 0x7fb85013f710 2024-03-25T18:31:04.102 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.101+0000 7fb8377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:04.102 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.101+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062469560 0 0) 0x7fb8501d7500 con 0x7fb8500a98f0 2024-03-25T18:31:04.102 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.101+0000 7fb8377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:04.102 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.101+0000 7fb8377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:04.103 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.101+0000 7fb8377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:04.103 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.101+0000 7fb8377fe640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:31:04.103 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.101+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb83c004eb0 con 0x7fb8500a98f0 2024-03-25T18:31:04.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.111+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391984272 0 0) 0x7fb8501d7830 con 0x7fb8501430c0 2024-03-25T18:31:04.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.111+0000 7fb8377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:04.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.111+0000 7fb8377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:04.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.111+0000 7fb8377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:04.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.111+0000 7fb8377fe640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:31:04.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.111+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8501d7500 con 0x7fb8501430c0 2024-03-25T18:31:04.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.114+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989038266 0 0) 0x7fb850137970 con 0x7fb85013f710 2024-03-25T18:31:04.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.114+0000 7fb8377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:04.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.114+0000 7fb8377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:04.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.114+0000 7fb8377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:04.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.114+0000 7fb8377fe640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:31:04.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.114+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8501d7830 con 0x7fb85013f710 2024-03-25T18:31:04.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.115+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2348746969 0 0) 0x7fb83c004eb0 con 0x7fb8500a98f0 2024-03-25T18:31:04.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.115+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb850137970 con 0x7fb8500a98f0 2024-03-25T18:31:04.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.115+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb8400019d0 con 0x7fb8500a98f0 2024-03-25T18:31:04.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.115+0000 7fb8377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:04.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.115+0000 7fb8377fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:04.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.115+0000 7fb8377fe640 10 monclient(hunting): dump: 2024-03-25T18:31:04.117 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:04.117 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:04.117 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:04.118 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:04.118 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:04.118 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:04.118 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:04.118 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:04.118 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:04.118 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:04.118 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.115+0000 7fb8377fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:04.127 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.125+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2979959521 0 0) 0x7fb850137970 con 0x7fb8500a98f0 2024-03-25T18:31:04.127 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.125+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 >> v1:172.21.15.99:6790/0 conn(0x7fb85013f710 legacy=0x7fb8501d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 >> v1:172.21.15.99:6789/0 conn(0x7fb8501430c0 legacy=0x7fb8501e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8377fe640 10 monclient: _finish_hunting 0 2024-03-25T18:31:04.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8377fe640 1 monclient: found mon.b 2024-03-25T18:31:04.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:04.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8377fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:04.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb850137b00 con 0x7fb8500a98f0 2024-03-25T18:31:04.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8377fe640 10 monclient: _finish_auth 0 2024-03-25T18:31:04.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8377fe640 10 monclient: _check_auth_tickets 2024-03-25T18:31:04.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:04.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8554d6640 5 monclient: authenticate success, global_id 4483 2024-03-25T18:31:04.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8554d6640 10 monclient: _renew_subs 2024-03-25T18:31:04.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8554d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:04.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8554d6640 1 -- 172.21.15.99:0/3007810447 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb850135080 con 0x7fb8500a98f0 2024-03-25T18:31:04.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8554d6640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:04.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8554d6640 10 monclient: _renew_subs 2024-03-25T18:31:04.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8554d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:04.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8554d6640 1 -- 172.21.15.99:0/3007810447 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8501e36d0 con 0x7fb8500a98f0 2024-03-25T18:31:04.129 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.126+0000 7fb8554d6640 1 librados: init done 2024-03-25T18:31:04.130 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.127+0000 7fb8554d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:04.130 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.127+0000 7fb8554d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:04.130 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.127+0000 7fb8554d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:04.130 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.127+0000 7fb8554d6640 1 -- 172.21.15.99:0/3007810447 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8501d7690 con 0x7fb8500a98f0 2024-03-25T18:31:04.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.399+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8400035f0 con 0x7fb8500a98f0 2024-03-25T18:31:04.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.399+0000 7fb8377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:04.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.399+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb840003860 con 0x7fb8500a98f0 2024-03-25T18:31:04.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.399+0000 7fb8377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:04.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.399+0000 7fb8377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:04.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.399+0000 7fb8377fe640 10 monclient: dump: 2024-03-25T18:31:04.402 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:04.402 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:04.402 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:04.402 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:04.402 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:04.402 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:04.402 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:04.402 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:04.402 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:04.403 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:04.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.399+0000 7fb84d7fa640 4 set_mon_vals no callback set 2024-03-25T18:31:04.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.401+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fb840003a80 con 0x7fb8500a98f0 2024-03-25T18:31:04.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.401+0000 7fb84e7fc640 10 monclient: get_auth_request con 0x7fb83c06ddf0 auth_method 0 2024-03-25T18:31:04.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.402+0000 7fb8377fe640 10 client.4483.objecter ms_handle_connect 0x7fb83c06ddf0 2024-03-25T18:31:04.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.405+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fb840004480 con 0x7fb8500a98f0 2024-03-25T18:31:04.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.405+0000 7fb8377fe640 10 client.4483.objecter ms_dispatch 0x7fb850000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:04.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.405+0000 7fb8377fe640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:04.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.405+0000 7fb8377fe640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:04.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.406+0000 7fb8377fe640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-25T18:31:04.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.406+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb840050150 con 0x7fb8500a98f0 2024-03-25T18:31:04.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.406+0000 7fb8377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:04.408 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.406+0000 7fb8377fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:04.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.655+0000 7fb8554d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:04.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.655+0000 7fb8554d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:04.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.655+0000 7fb8554d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:04.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.655+0000 7fb8554d6640 1 -- 172.21.15.99:0/3007810447 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8500627b0 con 0x7fb8500a98f0 2024-03-25T18:31:04.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.681+0000 7fb8377fe640 1 -- 172.21.15.99:0/3007810447 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fb840052080 con 0x7fb8500a98f0 2024-03-25T18:31:04.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.681+0000 7fb8377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:04.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.681+0000 7fb8377fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:04.682 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:31:04.683 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:04.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.683+0000 7fb8554d6640 10 librados: watch_flush enter 2024-03-25T18:31:04.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.683+0000 7fb8554d6640 10 librados: watch_flush exit 2024-03-25T18:31:04.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.683+0000 7fb8554d6640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-25T18:31:04.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.683+0000 7fb8554d6640 10 client.4483.objecter successfully canceled tick 2024-03-25T18:31:04.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.683+0000 7fb8554d6640 1 -- 172.21.15.99:0/3007810447 >> v1:172.21.15.190:6800/4220409699 conn(0x7fb83c06ddf0 legacy=0x7fb83c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.683+0000 7fb8554d6640 10 monclient: shutdown 2024-03-25T18:31:04.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.683+0000 7fb8554d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:04.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.683+0000 7fb8554d6640 1 -- 172.21.15.99:0/3007810447 >> v1:172.21.15.190:6789/0 conn(0x7fb8500a98f0 legacy=0x7fb850138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:04.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.683+0000 7fb8554d6640 1 -- 172.21.15.99:0/3007810447 shutdown_connections 2024-03-25T18:31:04.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.683+0000 7fb8554d6640 1 -- 172.21.15.99:0/3007810447 >> 172.21.15.99:0/3007810447 conn(0x7fb8500b7ea0 msgr2=0x7fb850074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:04.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.683+0000 7fb8554d6640 1 -- 172.21.15.99:0/3007810447 shutdown_connections 2024-03-25T18:31:04.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.683+0000 7fb8554d6640 1 -- 172.21.15.99:0/3007810447 wait complete. 2024-03-25T18:31:04.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:04.683+0000 7fb8554d6640 1 librados: shutdown 2024-03-25T18:31:04.697 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:09.418 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:06:10.010 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:09.424 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:06:10.016 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:09.425 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:06:10.018 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:09.461 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:06:10.053 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:09.472 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:06:10.030 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:09.485 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:06:10.043 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:09.490 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:06:10.083 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:09.511 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:06:10.069 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:09.525 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:06:10.118 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:11.504 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:11.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.619+0000 7fbcf13b5640 10 monclient: get_monmap_and_config 2024-03-25T18:31:11.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.619+0000 7fbcf13b5640 10 monclient: build_initial_monmap 2024-03-25T18:31:11.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.619+0000 7fbcf13b5640 10 monclient: monmap: 2024-03-25T18:31:11.620 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:11.620 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:11.620 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:31:11.620447+0000 2024-03-25T18:31:11.621 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:31:11.620447+0000 2024-03-25T18:31:11.621 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:11.621 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:11.621 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:11.621 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:31:11.621 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:31:11.621 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:11.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.620+0000 7fbcf13b5640 1 Processor -- start 2024-03-25T18:31:11.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.620+0000 7fbcf13b5640 1 -- start start 2024-03-25T18:31:11.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.620+0000 7fbcf13b5640 10 monclient: init 2024-03-25T18:31:11.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.620+0000 7fbcf13b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:11.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.620+0000 7fbcf13b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:11.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.620+0000 7fbcf13b5640 10 monclient(hunting): picked mon.noname-c con 0x7fbcec06ab80 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:11.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.621+0000 7fbcf13b5640 10 monclient(hunting): picked mon.noname-a con 0x7fbcec06af60 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:11.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.621+0000 7fbcf13b5640 10 monclient(hunting): picked mon.noname-b con 0x7fbcec0b00b0 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:11.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.621+0000 7fbcf13b5640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcec0b0500 con 0x7fbcec06af60 2024-03-25T18:31:11.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.621+0000 7fbcf13b5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcec13ffa0 con 0x7fbcec06ab80 2024-03-25T18:31:11.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.621+0000 7fbcf13b5640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcec0af520 con 0x7fbcec0b00b0 2024-03-25T18:31:11.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.621+0000 7fbcf13b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:11.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.621+0000 7fbcf13b5640 10 monclient(hunting): authenticate will time out at 7689.789041s 2024-03-25T18:31:11.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.630+0000 7fbceb7fe640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7fbcec0b00b0 0x7fbcec14dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:46030/0 (socket says 172.21.15.99:46030) 2024-03-25T18:31:11.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.630+0000 7fbceb7fe640 1 -- 172.21.15.99:0/2812398941 learned_addr learned my addr 172.21.15.99:0/2812398941 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:11.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.642+0000 7fbce9ffb640 1 -- 172.21.15.99:0/2812398941 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846467463 0 0) 0x7fbcec0af520 con 0x7fbcec0b00b0 2024-03-25T18:31:11.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.642+0000 7fbce9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:11.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.642+0000 7fbce9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:11.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.642+0000 7fbce9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:11.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.642+0000 7fbce9ffb640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:31:11.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.642+0000 7fbce9ffb640 1 -- 172.21.15.99:0/2812398941 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcd4003520 con 0x7fbcec0b00b0 2024-03-25T18:31:11.653 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.652+0000 7fbce9ffb640 1 -- 172.21.15.99:0/2812398941 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639622332 0 0) 0x7fbcec13ffa0 con 0x7fbcec06ab80 2024-03-25T18:31:11.653 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.652+0000 7fbce9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:11.653 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.652+0000 7fbce9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:11.654 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.652+0000 7fbce9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:11.654 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.652+0000 7fbce9ffb640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:31:11.654 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.652+0000 7fbce9ffb640 1 -- 172.21.15.99:0/2812398941 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcec0af520 con 0x7fbcec06ab80 2024-03-25T18:31:11.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 1 -- 172.21.15.99:0/2812398941 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 10929370 0 0) 0x7fbcd4003520 con 0x7fbcec0b00b0 2024-03-25T18:31:11.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 1 -- 172.21.15.99:0/2812398941 >> v1:172.21.15.190:6789/0 conn(0x7fbcec06ab80 legacy=0x7fbcec0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:11.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 1 -- 172.21.15.99:0/2812398941 >> v1:172.21.15.99:6789/0 conn(0x7fbcec06af60 legacy=0x7fbcec13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 1 monclient: found mon.noname-b 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 1 -- 172.21.15.99:0/2812398941 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec072890 con 0x7fbcec0b00b0 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 1 -- 172.21.15.99:0/2812398941 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbce4001a10 con 0x7fbcec0b00b0 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbce9ffb640 10 monclient: dump: 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:11.657 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:11.658 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:11.658 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:11.658 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:11.658 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:11.658 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:11.658 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:11.658 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:11.658 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbcf13b5640 5 monclient: authenticate success, global_id 4481 2024-03-25T18:31:11.658 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.655+0000 7fbcf13b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:11.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.668+0000 7fbce9ffb640 1 -- 172.21.15.99:0/2812398941 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbce4001b70 con 0x7fbcec0b00b0 2024-03-25T18:31:11.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.668+0000 7fbce9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:11.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.668+0000 7fbce9ffb640 1 -- 172.21.15.99:0/2812398941 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbce4001f80 con 0x7fbcec0b00b0 2024-03-25T18:31:11.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.668+0000 7fbce9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:11.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.668+0000 7fbce9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:11.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.668+0000 7fbce9ffb640 10 monclient: dump: 2024-03-25T18:31:11.670 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:11.670 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:11.670 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:11.670 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:11.670 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:11.670 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:11.670 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:11.670 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:11.670 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:11.670 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:11.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.668+0000 7fbcf13b5640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:11.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.668+0000 7fbcf13b5640 4 set_mon_vals no callback set 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.669+0000 7fbcf13b5640 10 monclient: shutdown 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.669+0000 7fbcf13b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.669+0000 7fbcf13b5640 1 -- 172.21.15.99:0/2812398941 >> v1:172.21.15.99:6790/0 conn(0x7fbcec0b00b0 legacy=0x7fbcec14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.669+0000 7fbcf13b5640 1 -- 172.21.15.99:0/2812398941 shutdown_connections 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.669+0000 7fbcf13b5640 1 -- 172.21.15.99:0/2812398941 >> 172.21.15.99:0/2812398941 conn(0x7fbcec0b4a80 msgr2=0x7fbcec0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.669+0000 7fbcf13b5640 1 -- 172.21.15.99:0/2812398941 shutdown_connections 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.669+0000 7fbcf13b5640 1 -- 172.21.15.99:0/2812398941 wait complete. 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.670+0000 7fbcf13b5640 10 monclient: build_initial_monmap 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.670+0000 7fbcf13b5640 10 monclient: monmap: 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:11.673 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:31:11.674 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:11.674 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:31:11.674 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:31:11.674 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:11.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.670+0000 7fbcf13b5640 1 librados: starting msgr at 2024-03-25T18:31:11.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.670+0000 7fbcf13b5640 1 librados: starting objecter 2024-03-25T18:31:11.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.670+0000 7fbcf13b5640 1 Processor -- start 2024-03-25T18:31:11.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.670+0000 7fbcf13b5640 1 -- start start 2024-03-25T18:31:11.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.670+0000 7fbcf13b5640 1 librados: setting wanted keys 2024-03-25T18:31:11.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.670+0000 7fbcf13b5640 1 librados: calling monclient init 2024-03-25T18:31:11.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.670+0000 7fbcf13b5640 10 monclient: init 2024-03-25T18:31:11.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.671+0000 7fbcf13b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:11.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.671+0000 7fbcf13b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:11.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.671+0000 7fbcf13b5640 10 monclient(hunting): picked mon.noname-c con 0x7fbcec06ab80 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:11.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.671+0000 7fbcf13b5640 10 monclient(hunting): picked mon.noname-a con 0x7fbcec0b00b0 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:11.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.671+0000 7fbcf13b5640 10 monclient(hunting): picked mon.noname-b con 0x7fbcec145ef0 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:11.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.671+0000 7fbcf13b5640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcec143460 con 0x7fbcec0b00b0 2024-03-25T18:31:11.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.671+0000 7fbcf13b5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcec145210 con 0x7fbcec145ef0 2024-03-25T18:31:11.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.671+0000 7fbcf13b5640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcec145480 con 0x7fbcec06ab80 2024-03-25T18:31:11.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.671+0000 7fbcf13b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:11.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.671+0000 7fbcf13b5640 10 monclient(hunting): authenticate will time out at 7689.839415s 2024-03-25T18:31:11.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.683+0000 7fbcea7fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fbcec0b00b0 0x7fbcec1d7550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51730/0 (socket says 172.21.15.99:51730) 2024-03-25T18:31:11.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.683+0000 7fbcea7fc640 1 -- 172.21.15.99:0/3481490648 learned_addr learned my addr 172.21.15.99:0/3481490648 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:11.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.684+0000 7fbcd37fe640 10 client.?.objecter ms_handle_connect 0x7fbcec0b00b0 2024-03-25T18:31:11.685 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.684+0000 7fbcd37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:11.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.685+0000 7fbcd37fe640 10 client.?.objecter ms_handle_connect 0x7fbcec06ab80 2024-03-25T18:31:11.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.685+0000 7fbcd37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:11.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.690+0000 7fbcd37fe640 10 client.?.objecter ms_handle_connect 0x7fbcec145ef0 2024-03-25T18:31:11.692 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.690+0000 7fbcd37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:11.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.693+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288828410 0 0) 0x7fbcec143460 con 0x7fbcec0b00b0 2024-03-25T18:31:11.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.693+0000 7fbcd37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:11.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.693+0000 7fbcd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:11.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.693+0000 7fbcd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:11.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.693+0000 7fbcd37fe640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:31:11.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.693+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcd8004eb0 con 0x7fbcec0b00b0 2024-03-25T18:31:11.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.706+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182130604 0 0) 0x7fbcec145210 con 0x7fbcec145ef0 2024-03-25T18:31:11.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.706+0000 7fbcd37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:11.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.706+0000 7fbcd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:11.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.706+0000 7fbcd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:11.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.706+0000 7fbcd37fe640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:31:11.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.706+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcec143460 con 0x7fbcec145ef0 2024-03-25T18:31:11.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.723+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3682294005 0 0) 0x7fbcd8004eb0 con 0x7fbcec0b00b0 2024-03-25T18:31:11.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.723+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcec145210 con 0x7fbcec0b00b0 2024-03-25T18:31:11.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.723+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbce0001a30 con 0x7fbcec0b00b0 2024-03-25T18:31:11.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.723+0000 7fbcd37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:11.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.723+0000 7fbcd37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:11.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.723+0000 7fbcd37fe640 10 monclient(hunting): dump: 2024-03-25T18:31:11.726 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:11.726 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:11.727 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:11.727 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:11.727 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:11.727 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:11.727 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:11.727 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:11.727 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:11.727 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:11.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.723+0000 7fbcd37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:11.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.727+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202779244 0 0) 0x7fbcec145480 con 0x7fbcec06ab80 2024-03-25T18:31:11.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.727+0000 7fbcd37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:11.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.727+0000 7fbcd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:11.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.727+0000 7fbcd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:11.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.727+0000 7fbcd37fe640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:31:11.729 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.727+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcd8004eb0 con 0x7fbcec06ab80 2024-03-25T18:31:11.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3748021755 0 0) 0x7fbcec145210 con 0x7fbcec0b00b0 2024-03-25T18:31:11.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 >> v1:172.21.15.99:6790/0 conn(0x7fbcec06ab80 legacy=0x7fbcec143f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:11.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 >> v1:172.21.15.190:6789/0 conn(0x7fbcec145ef0 legacy=0x7fbcec1d4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:11.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcd37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:31:11.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcd37fe640 1 monclient: found mon.a 2024-03-25T18:31:11.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:11.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcd37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:31:11.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec143c60 con 0x7fbcec0b00b0 2024-03-25T18:31:11.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcd37fe640 10 monclient: _finish_auth 0 2024-03-25T18:31:11.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcd37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:31:11.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:11.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcf13b5640 5 monclient: authenticate success, global_id 4473 2024-03-25T18:31:11.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcf13b5640 10 monclient: _renew_subs 2024-03-25T18:31:11.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcf13b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:31:11.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcf13b5640 1 -- 172.21.15.99:0/3481490648 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcec143090 con 0x7fbcec0b00b0 2024-03-25T18:31:11.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcf13b5640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:11.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcf13b5640 10 monclient: _renew_subs 2024-03-25T18:31:11.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcf13b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:31:11.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcf13b5640 1 -- 172.21.15.99:0/3481490648 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcec1d5100 con 0x7fbcec0b00b0 2024-03-25T18:31:11.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.734+0000 7fbcf13b5640 1 librados: init done 2024-03-25T18:31:11.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.735+0000 7fbcf13b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:11.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.735+0000 7fbcf13b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:11.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.735+0000 7fbcf13b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:31:11.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:11.735+0000 7fbcf13b5640 1 -- 172.21.15.99:0/3481490648 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcec0627b0 con 0x7fbcec0b00b0 2024-03-25T18:31:12.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.050+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbce0001c70 con 0x7fbcec0b00b0 2024-03-25T18:31:12.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.050+0000 7fbcd37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:12.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.050+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbce0003a60 con 0x7fbcec0b00b0 2024-03-25T18:31:12.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.050+0000 7fbcd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:12.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.050+0000 7fbcd37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:12.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.050+0000 7fbcd37fe640 10 monclient: dump: 2024-03-25T18:31:12.052 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:12.052 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:12.052 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:12.053 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:12.053 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:12.053 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:12.053 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:12.053 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:12.053 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:12.053 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:12.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.050+0000 7fbce97fa640 4 set_mon_vals no callback set 2024-03-25T18:31:12.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.051+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fbce0003c40 con 0x7fbcec0b00b0 2024-03-25T18:31:12.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.052+0000 7fbceaffd640 10 monclient: get_auth_request con 0x7fbcd806de20 auth_method 0 2024-03-25T18:31:12.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.052+0000 7fbcd37fe640 10 client.4473.objecter ms_handle_connect 0x7fbcd806de20 2024-03-25T18:31:12.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.056+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fbce00548b0 con 0x7fbcec0b00b0 2024-03-25T18:31:12.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.056+0000 7fbcd37fe640 10 client.4473.objecter ms_dispatch 0x7fbcec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:12.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.056+0000 7fbcd37fe640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:12.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.056+0000 7fbcd37fe640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:12.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.056+0000 7fbcd37fe640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-25T18:31:12.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.056+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbce00503c0 con 0x7fbcec0b00b0 2024-03-25T18:31:12.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.056+0000 7fbcd37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:12.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.056+0000 7fbcd37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:12.298 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.297+0000 7fbcf13b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:12.298 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.297+0000 7fbcf13b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:12.298 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.297+0000 7fbcf13b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:31:12.298 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.297+0000 7fbcf13b5640 1 -- 172.21.15.99:0/3481490648 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcec06fd80 con 0x7fbcec0b00b0 2024-03-25T18:31:12.329 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.328+0000 7fbcd37fe640 1 -- 172.21.15.99:0/3481490648 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fbce00522f0 con 0x7fbcec0b00b0 2024-03-25T18:31:12.329 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.328+0000 7fbcd37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:12.329 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.328+0000 7fbcd37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:12.329 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:31:12.329 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:12.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.330+0000 7fbcf13b5640 10 librados: watch_flush enter 2024-03-25T18:31:12.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.331+0000 7fbcf13b5640 10 librados: watch_flush exit 2024-03-25T18:31:12.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.331+0000 7fbcf13b5640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-25T18:31:12.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.331+0000 7fbcf13b5640 10 client.4473.objecter successfully canceled tick 2024-03-25T18:31:12.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.331+0000 7fbcf13b5640 1 -- 172.21.15.99:0/3481490648 >> v1:172.21.15.190:6800/4220409699 conn(0x7fbcd806de20 legacy=0x7fbcd80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:12.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.331+0000 7fbcf13b5640 10 monclient: shutdown 2024-03-25T18:31:12.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.331+0000 7fbcf13b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:12.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.331+0000 7fbcf13b5640 1 -- 172.21.15.99:0/3481490648 >> v1:172.21.15.99:6789/0 conn(0x7fbcec0b00b0 legacy=0x7fbcec1d7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:12.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.331+0000 7fbcf13b5640 1 -- 172.21.15.99:0/3481490648 shutdown_connections 2024-03-25T18:31:12.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.331+0000 7fbcf13b5640 1 -- 172.21.15.99:0/3481490648 >> 172.21.15.99:0/3481490648 conn(0x7fbcec0b4a80 msgr2=0x7fbcec06f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:12.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.331+0000 7fbcf13b5640 1 -- 172.21.15.99:0/3481490648 shutdown_connections 2024-03-25T18:31:12.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.331+0000 7fbcf13b5640 1 -- 172.21.15.99:0/3481490648 wait complete. 2024-03-25T18:31:12.333 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:12.331+0000 7fbcf13b5640 1 librados: shutdown 2024-03-25T18:31:12.343 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:19.150 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:19.266 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.265+0000 7f4facf23640 10 monclient: get_monmap_and_config 2024-03-25T18:31:19.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.265+0000 7f4facf23640 10 monclient: build_initial_monmap 2024-03-25T18:31:19.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.265+0000 7f4facf23640 10 monclient: monmap: 2024-03-25T18:31:19.267 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:19.267 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:19.267 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:31:19.266639+0000 2024-03-25T18:31:19.267 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:31:19.266639+0000 2024-03-25T18:31:19.267 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:19.267 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:19.267 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:19.267 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:31:19.267 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:31:19.267 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:19.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.266+0000 7f4facf23640 1 Processor -- start 2024-03-25T18:31:19.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.266+0000 7f4facf23640 1 -- start start 2024-03-25T18:31:19.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.266+0000 7f4facf23640 10 monclient: init 2024-03-25T18:31:19.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.267+0000 7f4facf23640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:19.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.267+0000 7f4facf23640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:19.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.267+0000 7f4facf23640 10 monclient(hunting): picked mon.noname-c con 0x7f4fa80b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:19.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.267+0000 7f4facf23640 10 monclient(hunting): picked mon.noname-a con 0x7f4fa80b1730 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:19.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.267+0000 7f4facf23640 10 monclient(hunting): picked mon.noname-b con 0x7f4fa814be90 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:19.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.267+0000 7f4facf23640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa80af940 con 0x7f4fa80b1730 2024-03-25T18:31:19.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.267+0000 7f4facf23640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa80b0a70 con 0x7f4fa80b4d60 2024-03-25T18:31:19.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.267+0000 7f4facf23640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa80b0d50 con 0x7f4fa814be90 2024-03-25T18:31:19.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.267+0000 7f4facf23640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:19.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.267+0000 7f4facf23640 10 monclient(hunting): authenticate will time out at 7697.435444s 2024-03-25T18:31:19.274 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.273+0000 7f4fa6d9d640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f4fa814be90 0x7f4fa814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:58706/0 (socket says 172.21.15.99:58706) 2024-03-25T18:31:19.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.273+0000 7f4fa6d9d640 1 -- 172.21.15.99:0/2054336824 learned_addr learned my addr 172.21.15.99:0/2054336824 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:19.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.282+0000 7f4fa5d9b640 1 -- 172.21.15.99:0/2054336824 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484945459 0 0) 0x7f4fa80b0d50 con 0x7f4fa814be90 2024-03-25T18:31:19.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.282+0000 7f4fa5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:19.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.282+0000 7f4fa5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:19.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.282+0000 7f4fa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:19.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.282+0000 7f4fa5d9b640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:31:19.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.282+0000 7f4fa5d9b640 1 -- 172.21.15.99:0/2054336824 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f94003520 con 0x7f4fa814be90 2024-03-25T18:31:19.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.285+0000 7f4fa5d9b640 1 -- 172.21.15.99:0/2054336824 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197085971 0 0) 0x7f4fa80b0a70 con 0x7f4fa80b4d60 2024-03-25T18:31:19.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.286+0000 7f4fa5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:19.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.286+0000 7f4fa5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:19.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.286+0000 7f4fa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:19.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.286+0000 7f4fa5d9b640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:31:19.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.286+0000 7f4fa5d9b640 1 -- 172.21.15.99:0/2054336824 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa80b0d50 con 0x7f4fa80b4d60 2024-03-25T18:31:19.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.291+0000 7f4fa5d9b640 1 -- 172.21.15.99:0/2054336824 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416038359 0 0) 0x7f4fa80af940 con 0x7f4fa80b1730 2024-03-25T18:31:19.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.291+0000 7f4fa5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:19.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.291+0000 7f4fa5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:19.293 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.291+0000 7f4fa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:19.293 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.291+0000 7f4fa5d9b640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:31:19.293 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.291+0000 7f4fa5d9b640 1 -- 172.21.15.99:0/2054336824 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa80b0a70 con 0x7f4fa80b1730 2024-03-25T18:31:19.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.293+0000 7f4fa5d9b640 1 -- 172.21.15.99:0/2054336824 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1509526368 0 0) 0x7f4f94003520 con 0x7f4fa814be90 2024-03-25T18:31:19.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.293+0000 7f4fa5d9b640 1 -- 172.21.15.99:0/2054336824 >> v1:172.21.15.190:6789/0 conn(0x7f4fa80b4d60 legacy=0x7f4fa813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:19.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.293+0000 7f4fa5d9b640 1 -- 172.21.15.99:0/2054336824 >> v1:172.21.15.99:6789/0 conn(0x7f4fa80b1730 legacy=0x7f4fa80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:19.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.293+0000 7f4fa5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:31:19.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.293+0000 7f4fa5d9b640 1 monclient: found mon.noname-b 2024-03-25T18:31:19.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.294+0000 7f4fa5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:19.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.294+0000 7f4fa5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:31:19.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.294+0000 7f4fa5d9b640 1 -- 172.21.15.99:0/2054336824 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa80b1030 con 0x7f4fa814be90 2024-03-25T18:31:19.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.294+0000 7f4fa5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:31:19.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.294+0000 7f4fa5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:31:19.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.294+0000 7f4fa5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:19.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.294+0000 7f4fa5d9b640 1 -- 172.21.15.99:0/2054336824 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4fa0001a10 con 0x7f4fa814be90 2024-03-25T18:31:19.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.294+0000 7f4fa5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:19.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.294+0000 7f4fa5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:19.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.294+0000 7f4fa5d9b640 10 monclient: dump: 2024-03-25T18:31:19.296 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:19.296 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:19.296 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:19.296 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:19.296 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:19.296 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:19.296 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:19.297 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:19.297 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:19.297 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:19.297 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.294+0000 7f4facf23640 5 monclient: authenticate success, global_id 4487 2024-03-25T18:31:19.297 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.294+0000 7f4facf23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:19.330 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.328+0000 7f4fa5d9b640 1 -- 172.21.15.99:0/2054336824 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fa0001b70 con 0x7f4fa814be90 2024-03-25T18:31:19.330 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.328+0000 7f4fa5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:19.330 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.328+0000 7f4fa5d9b640 1 -- 172.21.15.99:0/2054336824 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4fa0001f80 con 0x7f4fa814be90 2024-03-25T18:31:19.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.328+0000 7f4fa5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:19.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.328+0000 7f4fa5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:19.331 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.328+0000 7f4fa5d9b640 10 monclient: dump: 2024-03-25T18:31:19.331 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:19.331 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:19.331 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:19.331 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:19.331 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:19.331 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:19.331 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:19.331 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:19.332 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:19.332 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:19.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.328+0000 7f4facf23640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:19.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.328+0000 7f4facf23640 4 set_mon_vals no callback set 2024-03-25T18:31:19.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.329+0000 7f4facf23640 10 monclient: shutdown 2024-03-25T18:31:19.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.329+0000 7f4facf23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:19.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.329+0000 7f4facf23640 1 -- 172.21.15.99:0/2054336824 >> v1:172.21.15.99:6790/0 conn(0x7f4fa814be90 legacy=0x7f4fa814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:19.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.329+0000 7f4facf23640 1 -- 172.21.15.99:0/2054336824 shutdown_connections 2024-03-25T18:31:19.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.329+0000 7f4facf23640 1 -- 172.21.15.99:0/2054336824 >> 172.21.15.99:0/2054336824 conn(0x7f4fa80617a0 msgr2=0x7f4fa8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:19.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.329+0000 7f4facf23640 1 -- 172.21.15.99:0/2054336824 shutdown_connections 2024-03-25T18:31:19.332 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.329+0000 7f4facf23640 1 -- 172.21.15.99:0/2054336824 wait complete. 2024-03-25T18:31:19.334 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.330+0000 7f4facf23640 10 monclient: build_initial_monmap 2024-03-25T18:31:19.334 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.330+0000 7f4facf23640 10 monclient: monmap: 2024-03-25T18:31:19.334 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:19.334 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:19.334 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:31:19.334 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:31:19.335 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:19.335 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:31:19.335 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:19.335 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:31:19.335 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:31:19.335 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:19.335 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.330+0000 7f4facf23640 1 librados: starting msgr at 2024-03-25T18:31:19.335 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.330+0000 7f4facf23640 1 librados: starting objecter 2024-03-25T18:31:19.335 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.330+0000 7f4facf23640 1 Processor -- start 2024-03-25T18:31:19.335 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.331+0000 7f4facf23640 1 -- start start 2024-03-25T18:31:19.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.331+0000 7f4facf23640 1 librados: setting wanted keys 2024-03-25T18:31:19.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.331+0000 7f4facf23640 1 librados: calling monclient init 2024-03-25T18:31:19.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.331+0000 7f4facf23640 10 monclient: init 2024-03-25T18:31:19.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.331+0000 7f4facf23640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:19.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.331+0000 7f4facf23640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:19.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.331+0000 7f4facf23640 10 monclient(hunting): picked mon.noname-b con 0x7f4fa80b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:19.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.331+0000 7f4facf23640 10 monclient(hunting): picked mon.noname-a con 0x7f4fa80b4d60 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:19.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.331+0000 7f4facf23640 10 monclient(hunting): picked mon.noname-c con 0x7f4fa814be90 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:19.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.331+0000 7f4facf23640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa8141a50 con 0x7f4fa80b4d60 2024-03-25T18:31:19.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.331+0000 7f4facf23640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa8135810 con 0x7f4fa80b1730 2024-03-25T18:31:19.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.331+0000 7f4facf23640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa8141cc0 con 0x7f4fa814be90 2024-03-25T18:31:19.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.331+0000 7f4facf23640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:19.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.331+0000 7f4facf23640 10 monclient(hunting): authenticate will time out at 7697.499788s 2024-03-25T18:31:19.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.338+0000 7f4fa659c640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f4fa80b1730 0x7f4fa81dac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:39850/0 (socket says 172.21.15.99:39850) 2024-03-25T18:31:19.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.338+0000 7f4fa659c640 1 -- 172.21.15.99:0/514511417 learned_addr learned my addr 172.21.15.99:0/514511417 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:19.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.339+0000 7f4f9effd640 10 client.?.objecter ms_handle_connect 0x7f4fa80b1730 2024-03-25T18:31:19.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.339+0000 7f4f9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:19.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.349+0000 7f4f9effd640 10 client.?.objecter ms_handle_connect 0x7f4fa814be90 2024-03-25T18:31:19.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.349+0000 7f4f9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:19.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.350+0000 7f4f9effd640 10 client.?.objecter ms_handle_connect 0x7f4fa80b4d60 2024-03-25T18:31:19.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.350+0000 7f4f9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:19.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.352+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537917081 0 0) 0x7f4fa8135810 con 0x7f4fa80b1730 2024-03-25T18:31:19.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.352+0000 7f4f9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:19.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.352+0000 7f4f9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:19.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.352+0000 7f4f9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:19.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.352+0000 7f4f9effd640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:31:19.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.352+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f88004eb0 con 0x7f4fa80b1730 2024-03-25T18:31:19.363 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.362+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125706507 0 0) 0x7f4fa8141cc0 con 0x7f4fa814be90 2024-03-25T18:31:19.363 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.362+0000 7f4f9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:19.364 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.362+0000 7f4f9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:19.364 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.362+0000 7f4f9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:19.364 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.362+0000 7f4f9effd640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:31:19.364 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.362+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa8135810 con 0x7f4fa814be90 2024-03-25T18:31:19.364 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.363+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191942338 0 0) 0x7f4fa8141a50 con 0x7f4fa80b4d60 2024-03-25T18:31:19.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.363+0000 7f4f9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:19.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.363+0000 7f4f9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:19.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.363+0000 7f4f9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:19.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.363+0000 7f4f9effd640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:31:19.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.363+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa8141cc0 con 0x7f4fa80b4d60 2024-03-25T18:31:19.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.369+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1587902157 0 0) 0x7f4f88004eb0 con 0x7f4fa80b1730 2024-03-25T18:31:19.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.369+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fa8141a50 con 0x7f4fa80b1730 2024-03-25T18:31:19.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.369+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4f98001a30 con 0x7f4fa80b1730 2024-03-25T18:31:19.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.369+0000 7f4f9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:19.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.369+0000 7f4f9effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:19.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.369+0000 7f4f9effd640 10 monclient(hunting): dump: 2024-03-25T18:31:19.371 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:19.371 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:19.371 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:19.371 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:19.371 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:19.371 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:19.372 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:19.372 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:19.372 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:19.372 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:19.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.369+0000 7f4f9effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:19.381 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.379+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2037210282 0 0) 0x7f4fa8141cc0 con 0x7f4fa80b4d60 2024-03-25T18:31:19.381 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.379+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f88004eb0 con 0x7f4fa80b4d60 2024-03-25T18:31:19.381 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.379+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4f90001830 con 0x7f4fa80b4d60 2024-03-25T18:31:19.381 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.379+0000 7f4f9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:19.381 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.379+0000 7f4f9effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:19.381 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.379+0000 7f4f9effd640 10 monclient(hunting): dump: 2024-03-25T18:31:19.381 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:19.381 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:19.381 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:19.381 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:19.382 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:19.382 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:19.382 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:19.382 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:19.382 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:19.382 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:19.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.382+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1855544834 0 0) 0x7f4fa8135810 con 0x7f4fa814be90 2024-03-25T18:31:19.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.382+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fa8141cc0 con 0x7f4fa814be90 2024-03-25T18:31:19.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.382+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4fa0001e80 con 0x7f4fa814be90 2024-03-25T18:31:19.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.382+0000 7f4f9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:19.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.382+0000 7f4f9effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:19.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.382+0000 7f4f9effd640 10 monclient(hunting): dump: 2024-03-25T18:31:19.383 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:19.383 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:19.384 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:19.384 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:19.384 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:19.384 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:19.384 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:19.384 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:19.384 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:19.384 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:19.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 207303500 0 0) 0x7f4fa8141a50 con 0x7f4fa80b1730 2024-03-25T18:31:19.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 >> v1:172.21.15.99:6790/0 conn(0x7f4fa814be90 legacy=0x7f4fa8135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:19.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 >> v1:172.21.15.99:6789/0 conn(0x7f4fa80b4d60 legacy=0x7f4fa81349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:19.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4f9effd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:19.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4f9effd640 1 monclient: found mon.b 2024-03-25T18:31:19.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4f9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:19.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4f9effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:19.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa8140810 con 0x7f4fa80b1730 2024-03-25T18:31:19.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4f9effd640 10 monclient: _finish_auth 0 2024-03-25T18:31:19.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4f9effd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:19.392 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4f9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:19.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4facf23640 5 monclient: authenticate success, global_id 4495 2024-03-25T18:31:19.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4facf23640 10 monclient: _renew_subs 2024-03-25T18:31:19.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4facf23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:19.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4facf23640 1 -- 172.21.15.99:0/514511417 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa81377a0 con 0x7f4fa80b1730 2024-03-25T18:31:19.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4facf23640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:19.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4facf23640 10 monclient: _renew_subs 2024-03-25T18:31:19.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4facf23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:19.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4facf23640 1 -- 172.21.15.99:0/514511417 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa8140560 con 0x7f4fa80b1730 2024-03-25T18:31:19.393 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.390+0000 7f4facf23640 1 librados: init done 2024-03-25T18:31:19.394 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.391+0000 7f4facf23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:19.394 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.391+0000 7f4facf23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:19.394 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.391+0000 7f4facf23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:19.394 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.391+0000 7f4facf23640 1 -- 172.21.15.99:0/514511417 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa80627b0 con 0x7f4fa80b1730 2024-03-25T18:31:19.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.686+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f98001c70 con 0x7f4fa80b1730 2024-03-25T18:31:19.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.686+0000 7f4f9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:19.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.686+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4f98003a60 con 0x7f4fa80b1730 2024-03-25T18:31:19.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.686+0000 7f4f9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:19.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.686+0000 7f4f9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:19.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.686+0000 7f4f9effd640 10 monclient: dump: 2024-03-25T18:31:19.689 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:19.689 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:19.689 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:19.689 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:19.689 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:19.689 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:19.689 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:19.689 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:19.689 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:19.689 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:19.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.686+0000 7f4fa559a640 4 set_mon_vals no callback set 2024-03-25T18:31:19.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.687+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f4f98003c80 con 0x7f4fa80b1730 2024-03-25T18:31:19.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.688+0000 7f4f9ffff640 10 monclient: get_auth_request con 0x7f4f8806e0f0 auth_method 0 2024-03-25T18:31:19.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.688+0000 7f4f9effd640 10 client.4495.objecter ms_handle_connect 0x7f4f8806e0f0 2024-03-25T18:31:19.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.690+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f4f980548b0 con 0x7f4fa80b1730 2024-03-25T18:31:19.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.690+0000 7f4f9effd640 10 client.4495.objecter ms_dispatch 0x7f4fa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:19.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.690+0000 7f4f9effd640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:19.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.690+0000 7f4f9effd640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:19.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.690+0000 7f4f9effd640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.694+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f980503c0 con 0x7f4fa80b1730 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.694+0000 7f4f9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:19.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.694+0000 7f4f9effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:19.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.939+0000 7f4facf23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:19.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.939+0000 7f4facf23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:19.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.939+0000 7f4facf23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:19.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:19.939+0000 7f4facf23640 1 -- 172.21.15.99:0/514511417 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fa8069f40 con 0x7f4fa80b1730 2024-03-25T18:31:20.033 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.031+0000 7f4f9effd640 1 -- 172.21.15.99:0/514511417 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f4f980522f0 con 0x7f4fa80b1730 2024-03-25T18:31:20.033 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.031+0000 7f4f9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:20.033 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.031+0000 7f4f9effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:20.033 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:31:20.033 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:20.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.034+0000 7f4facf23640 10 librados: watch_flush enter 2024-03-25T18:31:20.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.034+0000 7f4facf23640 10 librados: watch_flush exit 2024-03-25T18:31:20.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.034+0000 7f4facf23640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-25T18:31:20.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.034+0000 7f4facf23640 10 client.4495.objecter successfully canceled tick 2024-03-25T18:31:20.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.034+0000 7f4facf23640 1 -- 172.21.15.99:0/514511417 >> v1:172.21.15.190:6800/4220409699 conn(0x7f4f8806e0f0 legacy=0x7f4f880705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.034+0000 7f4facf23640 10 monclient: shutdown 2024-03-25T18:31:20.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.034+0000 7f4facf23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:20.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.034+0000 7f4facf23640 1 -- 172.21.15.99:0/514511417 >> v1:172.21.15.190:6789/0 conn(0x7f4fa80b1730 legacy=0x7f4fa81dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.034+0000 7f4facf23640 1 -- 172.21.15.99:0/514511417 shutdown_connections 2024-03-25T18:31:20.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.034+0000 7f4facf23640 1 -- 172.21.15.99:0/514511417 >> 172.21.15.99:0/514511417 conn(0x7f4fa80617a0 msgr2=0x7f4fa813c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:20.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.034+0000 7f4facf23640 1 -- 172.21.15.99:0/514511417 shutdown_connections 2024-03-25T18:31:20.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.034+0000 7f4facf23640 1 -- 172.21.15.99:0/514511417 wait complete. 2024-03-25T18:31:20.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:20.034+0000 7f4facf23640 1 librados: shutdown 2024-03-25T18:31:20.046 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:26.853 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:26.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.963+0000 7f1924397640 10 monclient: get_monmap_and_config 2024-03-25T18:31:26.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.963+0000 7f1924397640 10 monclient: build_initial_monmap 2024-03-25T18:31:26.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.963+0000 7f1924397640 10 monclient: monmap: 2024-03-25T18:31:26.964 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:26.965 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:26.965 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:31:26.964624+0000 2024-03-25T18:31:26.965 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:31:26.964624+0000 2024-03-25T18:31:26.965 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:26.965 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:26.965 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:26.965 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:31:26.965 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:31:26.965 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:26.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.964+0000 7f1924397640 1 Processor -- start 2024-03-25T18:31:26.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.964+0000 7f1924397640 1 -- start start 2024-03-25T18:31:26.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.964+0000 7f1924397640 10 monclient: init 2024-03-25T18:31:26.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.965+0000 7f1924397640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:26.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.965+0000 7f1924397640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:26.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.965+0000 7f1924397640 10 monclient(hunting): picked mon.noname-b con 0x7f191c0a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:26.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.965+0000 7f1924397640 10 monclient(hunting): picked mon.noname-a con 0x7f191c13f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:26.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.965+0000 7f1924397640 10 monclient(hunting): picked mon.noname-c con 0x7f191c1430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:26.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.965+0000 7f1924397640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191c142520 con 0x7f191c13f710 2024-03-25T18:31:26.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.965+0000 7f1924397640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191c0b28a0 con 0x7f191c1430c0 2024-03-25T18:31:26.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.965+0000 7f1924397640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191c0b2a30 con 0x7f191c0a98f0 2024-03-25T18:31:26.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.965+0000 7f1924397640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:26.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.965+0000 7f1924397640 10 monclient(hunting): authenticate will time out at 7705.133485s 2024-03-25T18:31:26.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.971+0000 7f1922133640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f191c0a98f0 0x7f191c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:51730/0 (socket says 172.21.15.99:51730) 2024-03-25T18:31:26.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.971+0000 7f1922133640 1 -- 172.21.15.99:0/4284027139 learned_addr learned my addr 172.21.15.99:0/4284027139 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:26.985 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.983+0000 7f1921131640 1 -- 172.21.15.99:0/4284027139 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661192333 0 0) 0x7f191c0b2a30 con 0x7f191c0a98f0 2024-03-25T18:31:26.985 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.983+0000 7f1921131640 10 monclient(hunting): _negotiate 2024-03-25T18:31:26.985 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.983+0000 7f1921131640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:26.985 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.983+0000 7f1921131640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:26.985 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.983+0000 7f1921131640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:31:26.985 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.984+0000 7f1921131640 1 -- 172.21.15.99:0/4284027139 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1908003520 con 0x7f191c0a98f0 2024-03-25T18:31:26.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 1 -- 172.21.15.99:0/4284027139 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2818937633 0 0) 0x7f1908003520 con 0x7f191c0a98f0 2024-03-25T18:31:26.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 1 -- 172.21.15.99:0/4284027139 >> v1:172.21.15.190:6789/0 conn(0x7f191c1430c0 legacy=0x7f191c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:26.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 1 -- 172.21.15.99:0/4284027139 >> v1:172.21.15.99:6789/0 conn(0x7f191c13f710 legacy=0x7f191c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:26.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 10 monclient: _finish_hunting 0 2024-03-25T18:31:26.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 1 monclient: found mon.noname-b 2024-03-25T18:31:26.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:26.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:31:26.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 1 -- 172.21.15.99:0/4284027139 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c0b44c0 con 0x7f191c0a98f0 2024-03-25T18:31:26.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 10 monclient: _finish_auth 0 2024-03-25T18:31:26.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 10 monclient: _check_auth_tickets 2024-03-25T18:31:26.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 1 -- 172.21.15.99:0/4284027139 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f1918001690 con 0x7f191c0a98f0 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1921131640 10 monclient: dump: 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:26.998 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1924397640 5 monclient: authenticate success, global_id 4493 2024-03-25T18:31:26.999 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:26.995+0000 7f1924397640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:27.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.021+0000 7f1921131640 1 -- 172.21.15.99:0/4284027139 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19180017f0 con 0x7f191c0a98f0 2024-03-25T18:31:27.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.021+0000 7f1921131640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:27.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.021+0000 7f1921131640 1 -- 172.21.15.99:0/4284027139 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f1918001c40 con 0x7f191c0a98f0 2024-03-25T18:31:27.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.021+0000 7f1924397640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:27.023 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.021+0000 7f1924397640 4 set_mon_vals no callback set 2024-03-25T18:31:27.023 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.021+0000 7f1921131640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:27.023 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.021+0000 7f1921131640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:27.023 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.021+0000 7f1921131640 10 monclient: dump: 2024-03-25T18:31:27.023 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:27.023 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:27.023 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:27.023 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:27.023 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:27.023 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:27.023 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:27.023 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:27.023 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:27.023 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:27.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.022+0000 7f1924397640 10 monclient: shutdown 2024-03-25T18:31:27.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.022+0000 7f1924397640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:27.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.022+0000 7f1924397640 1 -- 172.21.15.99:0/4284027139 >> v1:172.21.15.99:6790/0 conn(0x7f191c0a98f0 legacy=0x7f191c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.022+0000 7f1924397640 1 -- 172.21.15.99:0/4284027139 shutdown_connections 2024-03-25T18:31:27.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.022+0000 7f1924397640 1 -- 172.21.15.99:0/4284027139 >> 172.21.15.99:0/4284027139 conn(0x7f191c0b7ea0 msgr2=0x7f191c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:27.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.022+0000 7f1924397640 1 -- 172.21.15.99:0/4284027139 shutdown_connections 2024-03-25T18:31:27.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.022+0000 7f1924397640 1 -- 172.21.15.99:0/4284027139 wait complete. 2024-03-25T18:31:27.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.023+0000 7f1924397640 10 monclient: build_initial_monmap 2024-03-25T18:31:27.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.023+0000 7f1924397640 10 monclient: monmap: 2024-03-25T18:31:27.026 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:27.026 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:27.026 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:31:27.026 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:31:27.026 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:27.026 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:31:27.026 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:27.026 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.023+0000 7f1924397640 1 librados: starting msgr at 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.023+0000 7f1924397640 1 librados: starting objecter 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.023+0000 7f1924397640 1 Processor -- start 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.023+0000 7f1924397640 1 -- start start 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.023+0000 7f1924397640 1 librados: setting wanted keys 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.023+0000 7f1924397640 1 librados: calling monclient init 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.023+0000 7f1924397640 10 monclient: init 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.024+0000 7f1924397640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.024+0000 7f1924397640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.024+0000 7f1924397640 10 monclient(hunting): picked mon.noname-a con 0x7f191c0a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.024+0000 7f1924397640 10 monclient(hunting): picked mon.noname-c con 0x7f191c13f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.024+0000 7f1924397640 10 monclient(hunting): picked mon.noname-b con 0x7f191c1430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.024+0000 7f1924397640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191c14e760 con 0x7f191c0a98f0 2024-03-25T18:31:27.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.024+0000 7f1924397640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191c14ea40 con 0x7f191c1430c0 2024-03-25T18:31:27.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.024+0000 7f1924397640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191c138d40 con 0x7f191c13f710 2024-03-25T18:31:27.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.024+0000 7f1924397640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:27.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.024+0000 7f1924397640 10 monclient(hunting): authenticate will time out at 7705.192480s 2024-03-25T18:31:27.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.030+0000 7f1921932640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f191c13f710 0x7f191c1da980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:51758/0 (socket says 172.21.15.99:51758) 2024-03-25T18:31:27.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.030+0000 7f1921932640 1 -- 172.21.15.99:0/3508859665 learned_addr learned my addr 172.21.15.99:0/3508859665 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:27.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.031+0000 7f1912ffd640 10 client.?.objecter ms_handle_connect 0x7f191c13f710 2024-03-25T18:31:27.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.031+0000 7f1912ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:27.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.034+0000 7f1912ffd640 10 client.?.objecter ms_handle_connect 0x7f191c1430c0 2024-03-25T18:31:27.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.034+0000 7f1912ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:27.046 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.045+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2961902875 0 0) 0x7f191c14ea40 con 0x7f191c1430c0 2024-03-25T18:31:27.046 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.045+0000 7f1912ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:27.046 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.045+0000 7f1912ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:27.046 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.045+0000 7f1912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:27.046 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.045+0000 7f1912ffd640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:31:27.046 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.045+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1900004eb0 con 0x7f191c1430c0 2024-03-25T18:31:27.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.048+0000 7f1912ffd640 10 client.?.objecter ms_handle_connect 0x7f191c0a98f0 2024-03-25T18:31:27.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.048+0000 7f1912ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:27.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.052+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727017105 0 0) 0x7f191c14e760 con 0x7f191c0a98f0 2024-03-25T18:31:27.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.052+0000 7f1912ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:27.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.052+0000 7f1912ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:27.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.052+0000 7f1912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:27.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.052+0000 7f1912ffd640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:31:27.054 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.052+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f191c14ea40 con 0x7f191c0a98f0 2024-03-25T18:31:27.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.056+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411106112 0 0) 0x7f191c138d40 con 0x7f191c13f710 2024-03-25T18:31:27.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.056+0000 7f1912ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:27.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.056+0000 7f1912ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:27.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.056+0000 7f1912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:27.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.056+0000 7f1912ffd640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:31:27.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.056+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f191c14e760 con 0x7f191c13f710 2024-03-25T18:31:27.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.061+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 991017891 0 0) 0x7f1900004eb0 con 0x7f191c1430c0 2024-03-25T18:31:27.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.061+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f191c138d40 con 0x7f191c1430c0 2024-03-25T18:31:27.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.061+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f1914001ae0 con 0x7f191c1430c0 2024-03-25T18:31:27.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.061+0000 7f1912ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:27.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.061+0000 7f1912ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:27.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.061+0000 7f1912ffd640 10 monclient(hunting): dump: 2024-03-25T18:31:27.062 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:27.062 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:27.062 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:27.062 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:27.063 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:27.063 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:27.063 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:27.063 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:27.063 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:27.063 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:27.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.061+0000 7f1912ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:27.065 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.064+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3115563652 0 0) 0x7f191c14ea40 con 0x7f191c0a98f0 2024-03-25T18:31:27.065 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.064+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1900004eb0 con 0x7f191c0a98f0 2024-03-25T18:31:27.065 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.064+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f1918001b00 con 0x7f191c0a98f0 2024-03-25T18:31:27.065 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.064+0000 7f1912ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:27.065 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.064+0000 7f1912ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:27.065 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.064+0000 7f1912ffd640 10 monclient(hunting): dump: 2024-03-25T18:31:27.065 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:27.066 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:27.066 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:27.066 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:27.066 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:27.066 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:27.066 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:27.066 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:27.066 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:27.066 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:27.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.068+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2329031846 0 0) 0x7f191c138d40 con 0x7f191c1430c0 2024-03-25T18:31:27.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.068+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 >> v1:172.21.15.99:6790/0 conn(0x7f191c13f710 legacy=0x7f191c1da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.068+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 >> v1:172.21.15.99:6789/0 conn(0x7f191c0a98f0 legacy=0x7f191c138630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.068+0000 7f1912ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:27.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.068+0000 7f1912ffd640 1 monclient: found mon.b 2024-03-25T18:31:27.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.068+0000 7f1912ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:27.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.068+0000 7f1912ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:27.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.068+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c138fe0 con 0x7f191c1430c0 2024-03-25T18:31:27.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.068+0000 7f1912ffd640 10 monclient: _finish_auth 0 2024-03-25T18:31:27.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.069+0000 7f1912ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:27.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.069+0000 7f1912ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:27.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.069+0000 7f1924397640 5 monclient: authenticate success, global_id 4501 2024-03-25T18:31:27.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.069+0000 7f1924397640 10 monclient: _renew_subs 2024-03-25T18:31:27.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.069+0000 7f1924397640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:27.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.069+0000 7f1924397640 1 -- 172.21.15.99:0/3508859665 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191c137a90 con 0x7f191c1430c0 2024-03-25T18:31:27.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.069+0000 7f1924397640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:27.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.069+0000 7f1924397640 10 monclient: _renew_subs 2024-03-25T18:31:27.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.069+0000 7f1924397640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:27.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.069+0000 7f1924397640 1 -- 172.21.15.99:0/3508859665 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f191c1e3020 con 0x7f191c1430c0 2024-03-25T18:31:27.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.069+0000 7f1924397640 1 librados: init done 2024-03-25T18:31:27.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.070+0000 7f1924397640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:27.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.070+0000 7f1924397640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:27.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.070+0000 7f1924397640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:27.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.070+0000 7f1924397640 1 -- 172.21.15.99:0/3508859665 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f191c0627b0 con 0x7f191c1430c0 2024-03-25T18:31:27.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.352+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1914002010 con 0x7f191c1430c0 2024-03-25T18:31:27.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.352+0000 7f1912ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:27.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.352+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f1914003b10 con 0x7f191c1430c0 2024-03-25T18:31:27.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.352+0000 7f1912ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:27.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.352+0000 7f1920930640 4 set_mon_vals no callback set 2024-03-25T18:31:27.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.352+0000 7f1912ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:27.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.352+0000 7f1912ffd640 10 monclient: dump: 2024-03-25T18:31:27.355 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:27.355 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:27.355 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:27.355 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:27.355 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:27.355 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:27.355 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:27.355 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:27.355 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:27.355 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:27.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.353+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f1914019ae0 con 0x7f191c1430c0 2024-03-25T18:31:27.356 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.353+0000 7f1922133640 10 monclient: get_auth_request con 0x7f190006df20 auth_method 0 2024-03-25T18:31:27.356 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.354+0000 7f1912ffd640 10 client.4501.objecter ms_handle_connect 0x7f190006df20 2024-03-25T18:31:27.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.357+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f19140537b0 con 0x7f191c1430c0 2024-03-25T18:31:27.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.357+0000 7f1912ffd640 10 client.4501.objecter ms_dispatch 0x7f191c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:27.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.357+0000 7f1912ffd640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:27.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.357+0000 7f1912ffd640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:27.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.357+0000 7f1912ffd640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-25T18:31:27.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.358+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f191404f2c0 con 0x7f191c1430c0 2024-03-25T18:31:27.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.358+0000 7f1912ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:27.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.358+0000 7f1912ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:27.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.608+0000 7f1924397640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:27.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.608+0000 7f1924397640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:27.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.608+0000 7f1924397640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:27.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.608+0000 7f1924397640 1 -- 172.21.15.99:0/3508859665 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f191c069f40 con 0x7f191c1430c0 2024-03-25T18:31:27.629 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.628+0000 7f1912ffd640 1 -- 172.21.15.99:0/3508859665 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f19140511f0 con 0x7f191c1430c0 2024-03-25T18:31:27.630 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.628+0000 7f1912ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:27.630 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.628+0000 7f1912ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:27.630 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:31:27.630 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:27.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.630+0000 7f1924397640 10 librados: watch_flush enter 2024-03-25T18:31:27.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.630+0000 7f1924397640 10 librados: watch_flush exit 2024-03-25T18:31:27.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.630+0000 7f1924397640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-25T18:31:27.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.630+0000 7f1924397640 10 client.4501.objecter successfully canceled tick 2024-03-25T18:31:27.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.630+0000 7f1924397640 1 -- 172.21.15.99:0/3508859665 >> v1:172.21.15.190:6800/4220409699 conn(0x7f190006df20 legacy=0x7f19000703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.630+0000 7f1924397640 10 monclient: shutdown 2024-03-25T18:31:27.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.630+0000 7f1924397640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:27.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.630+0000 7f1924397640 1 -- 172.21.15.99:0/3508859665 >> v1:172.21.15.190:6789/0 conn(0x7f191c1430c0 legacy=0x7f191c1db090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:27.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.630+0000 7f1924397640 1 -- 172.21.15.99:0/3508859665 shutdown_connections 2024-03-25T18:31:27.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.630+0000 7f1924397640 1 -- 172.21.15.99:0/3508859665 >> 172.21.15.99:0/3508859665 conn(0x7f191c0b7ea0 msgr2=0x7f191c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:27.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.631+0000 7f1924397640 1 -- 172.21.15.99:0/3508859665 shutdown_connections 2024-03-25T18:31:27.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.631+0000 7f1924397640 1 -- 172.21.15.99:0/3508859665 wait complete. 2024-03-25T18:31:27.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:27.631+0000 7f1924397640 1 librados: shutdown 2024-03-25T18:31:27.643 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:29.459 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:06:30.052 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:29.460 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:06:30.018 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:29.462 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:06:30.021 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:29.463 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:06:30.055 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:29.487 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:06:30.079 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:29.503 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:06:30.061 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:29.510 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:06:30.102 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:29.523 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:06:30.116 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:29.553 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:06:30.145 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:34.449 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:34.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.562+0000 7f0771967640 10 monclient: get_monmap_and_config 2024-03-25T18:31:34.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.562+0000 7f0771967640 10 monclient: build_initial_monmap 2024-03-25T18:31:34.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.562+0000 7f0771967640 10 monclient: monmap: 2024-03-25T18:31:34.563 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:34.563 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:34.563 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:31:34.563066+0000 2024-03-25T18:31:34.563 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:31:34.563066+0000 2024-03-25T18:31:34.563 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:34.563 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:34.563 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:34.564 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:31:34.564 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:31:34.564 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:34.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.562+0000 7f0771967640 1 Processor -- start 2024-03-25T18:31:34.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.562+0000 7f0771967640 1 -- start start 2024-03-25T18:31:34.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.562+0000 7f0771967640 10 monclient: init 2024-03-25T18:31:34.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.563+0000 7f0771967640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:34.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.563+0000 7f0771967640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:34.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.563+0000 7f0771967640 10 monclient(hunting): picked mon.noname-c con 0x7f076c0a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:34.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.563+0000 7f0771967640 10 monclient(hunting): picked mon.noname-b con 0x7f076c13f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:34.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.563+0000 7f0771967640 10 monclient(hunting): picked mon.noname-a con 0x7f076c1430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:34.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.563+0000 7f0771967640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c142520 con 0x7f076c1430c0 2024-03-25T18:31:34.565 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.563+0000 7f0771967640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c0b28a0 con 0x7f076c0a98f0 2024-03-25T18:31:34.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.563+0000 7f0771967640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c0b2a30 con 0x7f076c13f710 2024-03-25T18:31:34.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.563+0000 7f0771967640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:34.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.563+0000 7f0771967640 10 monclient(hunting): authenticate will time out at 7712.731348s 2024-03-25T18:31:34.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.577+0000 7f076b7fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f076c1430c0 0x7f076c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40680/0 (socket says 172.21.15.99:40680) 2024-03-25T18:31:34.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.577+0000 7f076b7fe640 1 -- 172.21.15.99:0/2759786202 learned_addr learned my addr 172.21.15.99:0/2759786202 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:34.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.597+0000 7f0769ffb640 1 -- 172.21.15.99:0/2759786202 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910647860 0 0) 0x7f076c0b28a0 con 0x7f076c0a98f0 2024-03-25T18:31:34.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.597+0000 7f0769ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:34.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.597+0000 7f0769ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:34.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.597+0000 7f0769ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:34.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.597+0000 7f0769ffb640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:31:34.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.597+0000 7f0769ffb640 1 -- 172.21.15.99:0/2759786202 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0754003520 con 0x7f076c0a98f0 2024-03-25T18:31:34.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.598+0000 7f0769ffb640 1 -- 172.21.15.99:0/2759786202 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948052007 0 0) 0x7f076c142520 con 0x7f076c1430c0 2024-03-25T18:31:34.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.598+0000 7f0769ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:34.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.598+0000 7f0769ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:34.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.598+0000 7f0769ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:34.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.598+0000 7f0769ffb640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:31:34.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.598+0000 7f0769ffb640 1 -- 172.21.15.99:0/2759786202 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f076c0b28a0 con 0x7f076c1430c0 2024-03-25T18:31:34.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.600+0000 7f0769ffb640 1 -- 172.21.15.99:0/2759786202 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914207192 0 0) 0x7f076c0b2a30 con 0x7f076c13f710 2024-03-25T18:31:34.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.600+0000 7f0769ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:34.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.600+0000 7f0769ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:34.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.600+0000 7f0769ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:34.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.600+0000 7f0769ffb640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:31:34.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.600+0000 7f0769ffb640 1 -- 172.21.15.99:0/2759786202 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f076c142520 con 0x7f076c13f710 2024-03-25T18:31:34.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 1 -- 172.21.15.99:0/2759786202 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3930827548 0 0) 0x7f076c142520 con 0x7f076c13f710 2024-03-25T18:31:34.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 1 -- 172.21.15.99:0/2759786202 >> v1:172.21.15.190:6789/0 conn(0x7f076c0a98f0 legacy=0x7f076c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:34.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 1 -- 172.21.15.99:0/2759786202 >> v1:172.21.15.99:6789/0 conn(0x7f076c1430c0 legacy=0x7f076c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:34.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:34.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 1 monclient: found mon.noname-b 2024-03-25T18:31:34.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:34.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:31:34.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 1 -- 172.21.15.99:0/2759786202 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c0b44c0 con 0x7f076c13f710 2024-03-25T18:31:34.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:34.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:34.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:34.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 1 -- 172.21.15.99:0/2759786202 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f07600015b0 con 0x7f076c13f710 2024-03-25T18:31:34.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:34.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:34.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0769ffb640 10 monclient: dump: 2024-03-25T18:31:34.615 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:34.615 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:34.615 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:34.615 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:34.615 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:34.615 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:34.615 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:34.615 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:34.615 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:34.615 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:34.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0771967640 5 monclient: authenticate success, global_id 4499 2024-03-25T18:31:34.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.612+0000 7f0771967640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:34.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.620+0000 7f0769ffb640 1 -- 172.21.15.99:0/2759786202 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0760001710 con 0x7f076c13f710 2024-03-25T18:31:34.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.620+0000 7f0769ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:34.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.620+0000 7f0769ffb640 1 -- 172.21.15.99:0/2759786202 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0760001b60 con 0x7f076c13f710 2024-03-25T18:31:34.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.620+0000 7f0771967640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:34.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.620+0000 7f0771967640 4 set_mon_vals no callback set 2024-03-25T18:31:34.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.620+0000 7f0769ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:34.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.620+0000 7f0769ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:34.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.620+0000 7f0769ffb640 10 monclient: dump: 2024-03-25T18:31:34.622 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:34.622 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:34.622 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:34.622 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:34.622 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:34.622 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:34.622 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:34.622 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:34.622 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:34.622 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:34.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.620+0000 7f0771967640 10 monclient: shutdown 2024-03-25T18:31:34.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.621+0000 7f0771967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:34.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.621+0000 7f0771967640 1 -- 172.21.15.99:0/2759786202 >> v1:172.21.15.99:6790/0 conn(0x7f076c13f710 legacy=0x7f076c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:34.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.621+0000 7f0771967640 1 -- 172.21.15.99:0/2759786202 shutdown_connections 2024-03-25T18:31:34.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.621+0000 7f0771967640 1 -- 172.21.15.99:0/2759786202 >> 172.21.15.99:0/2759786202 conn(0x7f076c0b7ea0 msgr2=0x7f076c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:34.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.621+0000 7f0771967640 1 -- 172.21.15.99:0/2759786202 shutdown_connections 2024-03-25T18:31:34.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.621+0000 7f0771967640 1 -- 172.21.15.99:0/2759786202 wait complete. 2024-03-25T18:31:34.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.621+0000 7f0771967640 10 monclient: build_initial_monmap 2024-03-25T18:31:34.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.621+0000 7f0771967640 10 monclient: monmap: 2024-03-25T18:31:34.624 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:34.625 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:34.625 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:31:34.625 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:31:34.625 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:34.625 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:31:34.625 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:34.625 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:31:34.625 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:31:34.625 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:34.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.622+0000 7f0771967640 1 librados: starting msgr at 2024-03-25T18:31:34.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.622+0000 7f0771967640 1 librados: starting objecter 2024-03-25T18:31:34.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.622+0000 7f0771967640 1 Processor -- start 2024-03-25T18:31:34.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.622+0000 7f0771967640 1 -- start start 2024-03-25T18:31:34.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.622+0000 7f0771967640 1 librados: setting wanted keys 2024-03-25T18:31:34.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.622+0000 7f0771967640 1 librados: calling monclient init 2024-03-25T18:31:34.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.622+0000 7f0771967640 10 monclient: init 2024-03-25T18:31:34.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.622+0000 7f0771967640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:34.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.622+0000 7f0771967640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:34.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.622+0000 7f0771967640 10 monclient(hunting): picked mon.noname-a con 0x7f076c0a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:34.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.623+0000 7f0771967640 10 monclient(hunting): picked mon.noname-c con 0x7f076c1430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:34.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.623+0000 7f0771967640 10 monclient(hunting): picked mon.noname-b con 0x7f076c135370 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:34.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.623+0000 7f0771967640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c13bd00 con 0x7f076c0a98f0 2024-03-25T18:31:34.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.623+0000 7f0771967640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c137010 con 0x7f076c135370 2024-03-25T18:31:34.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.623+0000 7f0771967640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c1309a0 con 0x7f076c1430c0 2024-03-25T18:31:34.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.623+0000 7f0771967640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:34.627 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.623+0000 7f0771967640 10 monclient(hunting): authenticate will time out at 7712.790942s 2024-03-25T18:31:34.640 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.639+0000 7f076b7fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f076c135370 0x7f076c130290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:36058/0 (socket says 172.21.15.99:36058) 2024-03-25T18:31:34.640 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.639+0000 7f076b7fe640 1 -- 172.21.15.99:0/1708460321 learned_addr learned my addr 172.21.15.99:0/1708460321 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:34.640 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.639+0000 7f0753fff640 10 client.?.objecter ms_handle_connect 0x7f076c135370 2024-03-25T18:31:34.641 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.639+0000 7f0753fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:34.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.644+0000 7f0753fff640 10 client.?.objecter ms_handle_connect 0x7f076c1430c0 2024-03-25T18:31:34.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.644+0000 7f0753fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:34.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.647+0000 7f0753fff640 10 client.?.objecter ms_handle_connect 0x7f076c0a98f0 2024-03-25T18:31:34.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.647+0000 7f0753fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:34.651 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.650+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307846730 0 0) 0x7f076c137010 con 0x7f076c135370 2024-03-25T18:31:34.651 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.650+0000 7f0753fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:34.651 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.650+0000 7f0753fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:34.652 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.650+0000 7f0753fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:34.652 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.650+0000 7f0753fff640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:31:34.652 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.650+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0758004eb0 con 0x7f076c135370 2024-03-25T18:31:34.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.654+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190105689 0 0) 0x7f076c13bd00 con 0x7f076c0a98f0 2024-03-25T18:31:34.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.654+0000 7f0753fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:34.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.654+0000 7f0753fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:34.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.654+0000 7f0753fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:34.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.654+0000 7f0753fff640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:31:34.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.654+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f076c137010 con 0x7f076c0a98f0 2024-03-25T18:31:34.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.654+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2072357958 0 0) 0x7f076c1309a0 con 0x7f076c1430c0 2024-03-25T18:31:34.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.654+0000 7f0753fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:34.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.654+0000 7f0753fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:34.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.654+0000 7f0753fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:34.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.654+0000 7f0753fff640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:31:34.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.654+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f076c13bd00 con 0x7f076c1430c0 2024-03-25T18:31:34.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.664+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1366919279 0 0) 0x7f076c13bd00 con 0x7f076c1430c0 2024-03-25T18:31:34.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.664+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f076c1309a0 con 0x7f076c1430c0 2024-03-25T18:31:34.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.664+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0760001a20 con 0x7f076c1430c0 2024-03-25T18:31:34.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.664+0000 7f0753fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:34.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.664+0000 7f0753fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:34.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.664+0000 7f0753fff640 10 monclient(hunting): dump: 2024-03-25T18:31:34.666 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:34.666 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:34.666 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:34.666 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:34.666 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:34.666 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:34.666 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:34.666 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:34.667 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:34.667 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:34.667 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.664+0000 7f0753fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:34.667 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.666+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 710916891 0 0) 0x7f076c137010 con 0x7f076c0a98f0 2024-03-25T18:31:34.667 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.666+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f076c13bd00 con 0x7f076c0a98f0 2024-03-25T18:31:34.667 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.666+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f075c001cd0 con 0x7f076c0a98f0 2024-03-25T18:31:34.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.666+0000 7f0753fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:34.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.666+0000 7f0753fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:34.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.666+0000 7f0753fff640 10 monclient(hunting): dump: 2024-03-25T18:31:34.668 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:34.668 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:34.668 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:34.668 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:34.668 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:34.668 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:34.668 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:34.668 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:34.668 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:34.668 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:34.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.672+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2820599500 0 0) 0x7f0758004eb0 con 0x7f076c135370 2024-03-25T18:31:34.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.672+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f076c137010 con 0x7f076c135370 2024-03-25T18:31:34.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.672+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0764001890 con 0x7f076c135370 2024-03-25T18:31:34.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.672+0000 7f0753fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:34.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.672+0000 7f0753fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:34.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.672+0000 7f0753fff640 10 monclient(hunting): dump: 2024-03-25T18:31:34.675 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:34.675 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:34.675 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:34.675 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:34.675 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:34.675 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:34.675 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:34.675 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:34.675 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:34.675 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:34.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 180107359 0 0) 0x7f076c13bd00 con 0x7f076c0a98f0 2024-03-25T18:31:34.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 >> v1:172.21.15.99:6790/0 conn(0x7f076c1430c0 legacy=0x7f076c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:34.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 >> v1:172.21.15.190:6789/0 conn(0x7f076c135370 legacy=0x7f076c130290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:34.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0753fff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:34.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0753fff640 1 monclient: found mon.a 2024-03-25T18:31:34.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0753fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:34.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0753fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:31:34.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c1363e0 con 0x7f076c0a98f0 2024-03-25T18:31:34.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0753fff640 10 monclient: _finish_auth 0 2024-03-25T18:31:34.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0753fff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:34.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0753fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:34.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0771967640 5 monclient: authenticate success, global_id 4491 2024-03-25T18:31:34.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0771967640 10 monclient: _renew_subs 2024-03-25T18:31:34.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0771967640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:31:34.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0771967640 1 -- 172.21.15.99:0/1708460321 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076c137280 con 0x7f076c0a98f0 2024-03-25T18:31:34.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0771967640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:34.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0771967640 10 monclient: _renew_subs 2024-03-25T18:31:34.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0771967640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:31:34.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0771967640 1 -- 172.21.15.99:0/1708460321 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f076c135970 con 0x7f076c0a98f0 2024-03-25T18:31:34.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.677+0000 7f0771967640 1 librados: init done 2024-03-25T18:31:34.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.679+0000 7f0771967640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:34.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.679+0000 7f0771967640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:34.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.679+0000 7f0771967640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:31:34.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.679+0000 7f0771967640 1 -- 172.21.15.99:0/1708460321 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f076c0b3ef0 con 0x7f076c0a98f0 2024-03-25T18:31:34.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.975+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f075c0020e0 con 0x7f076c0a98f0 2024-03-25T18:31:34.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.975+0000 7f0753fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:34.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.975+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f075c003b10 con 0x7f076c0a98f0 2024-03-25T18:31:34.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.975+0000 7f0753fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:34.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.975+0000 7f0753fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:34.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.975+0000 7f0753fff640 10 monclient: dump: 2024-03-25T18:31:34.976 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:34.976 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:34.976 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:34.977 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:34.977 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:34.977 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:34.977 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:34.977 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:34.977 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:34.977 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:34.977 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.975+0000 7f07697fa640 4 set_mon_vals no callback set 2024-03-25T18:31:34.978 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.976+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f075c019ae0 con 0x7f076c0a98f0 2024-03-25T18:31:34.978 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.976+0000 7f076a7fc640 10 monclient: get_auth_request con 0x7f075806e110 auth_method 0 2024-03-25T18:31:34.978 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.977+0000 7f0753fff640 10 client.4491.objecter ms_handle_connect 0x7f075806e110 2024-03-25T18:31:34.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.980+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f075c053840 con 0x7f076c0a98f0 2024-03-25T18:31:34.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.980+0000 7f0753fff640 10 client.4491.objecter ms_dispatch 0x7f076c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:34.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.980+0000 7f0753fff640 3 client.4491.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:34.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.980+0000 7f0753fff640 3 client.4491.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:34.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.980+0000 7f0753fff640 20 client.4491.objecter dump_active .. 0 homeless 2024-03-25T18:31:34.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.980+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f075c04f180 con 0x7f076c0a98f0 2024-03-25T18:31:34.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.980+0000 7f0753fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:34.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:34.980+0000 7f0753fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:35.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.212+0000 7f0771967640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:35.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.212+0000 7f0771967640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:35.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.212+0000 7f0771967640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:31:35.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.212+0000 7f0771967640 1 -- 172.21.15.99:0/1708460321 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f076c14f320 con 0x7f076c0a98f0 2024-03-25T18:31:35.242 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.240+0000 7f0753fff640 1 -- 172.21.15.99:0/1708460321 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f075c051280 con 0x7f076c0a98f0 2024-03-25T18:31:35.242 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.240+0000 7f0753fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:35.242 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.240+0000 7f0753fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:35.242 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:31:35.242 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:35.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.243+0000 7f0771967640 10 librados: watch_flush enter 2024-03-25T18:31:35.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.243+0000 7f0771967640 10 librados: watch_flush exit 2024-03-25T18:31:35.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.243+0000 7f0771967640 20 client.4491.objecter shutdown clearing up homeless session... 2024-03-25T18:31:35.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.243+0000 7f0771967640 10 client.4491.objecter successfully canceled tick 2024-03-25T18:31:35.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.243+0000 7f0771967640 1 -- 172.21.15.99:0/1708460321 >> v1:172.21.15.190:6800/4220409699 conn(0x7f075806e110 legacy=0x7f07580705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.243+0000 7f0771967640 10 monclient: shutdown 2024-03-25T18:31:35.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.243+0000 7f0771967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:35.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.243+0000 7f0771967640 1 -- 172.21.15.99:0/1708460321 >> v1:172.21.15.99:6789/0 conn(0x7f076c0a98f0 legacy=0x7f076c12fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.243+0000 7f0771967640 1 -- 172.21.15.99:0/1708460321 shutdown_connections 2024-03-25T18:31:35.246 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.243+0000 7f0771967640 1 -- 172.21.15.99:0/1708460321 >> 172.21.15.99:0/1708460321 conn(0x7f076c0b7ea0 msgr2=0x7f076c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:35.246 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.243+0000 7f0771967640 1 -- 172.21.15.99:0/1708460321 shutdown_connections 2024-03-25T18:31:35.246 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.243+0000 7f0771967640 1 -- 172.21.15.99:0/1708460321 wait complete. 2024-03-25T18:31:35.246 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:35.243+0000 7f0771967640 1 librados: shutdown 2024-03-25T18:31:35.256 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:42.062 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:42.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.180+0000 7facc9e05640 10 monclient: get_monmap_and_config 2024-03-25T18:31:42.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.180+0000 7facc9e05640 10 monclient: build_initial_monmap 2024-03-25T18:31:42.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.180+0000 7facc9e05640 10 monclient: monmap: 2024-03-25T18:31:42.182 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:42.182 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:42.182 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:31:42.181904+0000 2024-03-25T18:31:42.182 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:31:42.181904+0000 2024-03-25T18:31:42.182 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:42.183 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:42.183 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:42.183 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:31:42.183 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:31:42.183 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:42.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.181+0000 7facc9e05640 1 Processor -- start 2024-03-25T18:31:42.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.181+0000 7facc9e05640 1 -- start start 2024-03-25T18:31:42.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.181+0000 7facc9e05640 10 monclient: init 2024-03-25T18:31:42.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.182+0000 7facc9e05640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:42.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.182+0000 7facc9e05640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:31:42.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.182+0000 7facc9e05640 10 monclient(hunting): picked mon.noname-b con 0x7facc40a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:42.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.182+0000 7facc9e05640 10 monclient(hunting): picked mon.noname-c con 0x7facc413f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:42.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.182+0000 7facc9e05640 10 monclient(hunting): picked mon.noname-a con 0x7facc41430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:42.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.182+0000 7facc9e05640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc4142520 con 0x7facc41430c0 2024-03-25T18:31:42.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.182+0000 7facc9e05640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc40b28a0 con 0x7facc413f710 2024-03-25T18:31:42.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.182+0000 7facc9e05640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc40b2a30 con 0x7facc40a98f0 2024-03-25T18:31:42.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.182+0000 7facc9e05640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:42.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.182+0000 7facc9e05640 10 monclient(hunting): authenticate will time out at 7720.350750s 2024-03-25T18:31:42.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.189+0000 7facc37fe640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7facc40a98f0 0x7facc40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:43052/0 (socket says 172.21.15.99:43052) 2024-03-25T18:31:42.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.189+0000 7facc37fe640 1 -- 172.21.15.99:0/2662485202 learned_addr learned my addr 172.21.15.99:0/2662485202 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:42.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.201+0000 7facc27fc640 1 -- 172.21.15.99:0/2662485202 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832259003 0 0) 0x7facc40b2a30 con 0x7facc40a98f0 2024-03-25T18:31:42.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.201+0000 7facc27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:42.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.201+0000 7facc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:42.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.201+0000 7facc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:42.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.201+0000 7facc27fc640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:31:42.203 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.201+0000 7facc27fc640 1 -- 172.21.15.99:0/2662485202 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facac003520 con 0x7facc40a98f0 2024-03-25T18:31:42.209 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.208+0000 7facc27fc640 1 -- 172.21.15.99:0/2662485202 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211521530 0 0) 0x7facc40b28a0 con 0x7facc413f710 2024-03-25T18:31:42.209 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.208+0000 7facc27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:42.209 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.208+0000 7facc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:42.210 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.208+0000 7facc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:42.210 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.208+0000 7facc27fc640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:31:42.210 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.208+0000 7facc27fc640 1 -- 172.21.15.99:0/2662485202 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc40b2a30 con 0x7facc413f710 2024-03-25T18:31:42.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 1 -- 172.21.15.99:0/2662485202 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3412718951 0 0) 0x7facc40b2a30 con 0x7facc413f710 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 1 -- 172.21.15.99:0/2662485202 >> v1:172.21.15.99:6790/0 conn(0x7facc40a98f0 legacy=0x7facc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 1 -- 172.21.15.99:0/2662485202 >> v1:172.21.15.99:6789/0 conn(0x7facc41430c0 legacy=0x7facc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 1 monclient: found mon.noname-c 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 1 -- 172.21.15.99:0/2662485202 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc40b44c0 con 0x7facc413f710 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 10 monclient: _finish_auth 0 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 1 -- 172.21.15.99:0/2662485202 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7facb8001670 con 0x7facc413f710 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc27fc640 10 monclient: dump: 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:42.215 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:42.216 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:42.216 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:42.216 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:42.216 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:42.216 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:42.216 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:42.216 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:42.216 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:42.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc9e05640 5 monclient: authenticate success, global_id 4510 2024-03-25T18:31:42.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.213+0000 7facc9e05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:42.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.222+0000 7facc27fc640 1 -- 172.21.15.99:0/2662485202 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facb8001890 con 0x7facc413f710 2024-03-25T18:31:42.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.222+0000 7facc27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:42.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.222+0000 7facc27fc640 1 -- 172.21.15.99:0/2662485202 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7facb8001ba0 con 0x7facc413f710 2024-03-25T18:31:42.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.222+0000 7facc9e05640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:42.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.222+0000 7facc9e05640 4 set_mon_vals no callback set 2024-03-25T18:31:42.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.222+0000 7facc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:42.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.222+0000 7facc27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:42.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.222+0000 7facc27fc640 10 monclient: dump: 2024-03-25T18:31:42.223 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:42.223 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:42.223 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:42.223 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:42.224 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:42.224 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:42.224 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:42.224 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:42.224 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:42.224 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:42.225 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.222+0000 7facc9e05640 10 monclient: shutdown 2024-03-25T18:31:42.225 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.222+0000 7facc9e05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:42.225 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.222+0000 7facc9e05640 1 -- 172.21.15.99:0/2662485202 >> v1:172.21.15.190:6789/0 conn(0x7facc413f710 legacy=0x7facc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:42.225 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.223+0000 7facc9e05640 1 -- 172.21.15.99:0/2662485202 shutdown_connections 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.223+0000 7facc9e05640 1 -- 172.21.15.99:0/2662485202 >> 172.21.15.99:0/2662485202 conn(0x7facc40b7ea0 msgr2=0x7facc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.223+0000 7facc9e05640 1 -- 172.21.15.99:0/2662485202 shutdown_connections 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.223+0000 7facc9e05640 1 -- 172.21.15.99:0/2662485202 wait complete. 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.223+0000 7facc9e05640 10 monclient: build_initial_monmap 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.223+0000 7facc9e05640 10 monclient: monmap: 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:31:42.226 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:42.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.224+0000 7facc9e05640 1 librados: starting msgr at 2024-03-25T18:31:42.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.224+0000 7facc9e05640 1 librados: starting objecter 2024-03-25T18:31:42.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.224+0000 7facc9e05640 1 Processor -- start 2024-03-25T18:31:42.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.224+0000 7facc9e05640 1 -- start start 2024-03-25T18:31:42.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.224+0000 7facc9e05640 1 librados: setting wanted keys 2024-03-25T18:31:42.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.224+0000 7facc9e05640 1 librados: calling monclient init 2024-03-25T18:31:42.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.224+0000 7facc9e05640 10 monclient: init 2024-03-25T18:31:42.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.225+0000 7facc9e05640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:42.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.225+0000 7facc9e05640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:42.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.225+0000 7facc9e05640 10 monclient(hunting): picked mon.noname-a con 0x7facc40a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:42.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.225+0000 7facc9e05640 10 monclient(hunting): picked mon.noname-c con 0x7facc413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:42.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.225+0000 7facc9e05640 10 monclient(hunting): picked mon.noname-b con 0x7facc41430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:42.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.225+0000 7facc9e05640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc40ab9c0 con 0x7facc40a98f0 2024-03-25T18:31:42.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.225+0000 7facc9e05640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc41db780 con 0x7facc41430c0 2024-03-25T18:31:42.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.225+0000 7facc9e05640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc4139990 con 0x7facc413f710 2024-03-25T18:31:42.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.225+0000 7facc9e05640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:42.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.225+0000 7facc9e05640 10 monclient(hunting): authenticate will time out at 7720.393520s 2024-03-25T18:31:42.242 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.241+0000 7facc3fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7facc41430c0 0x7facc41e1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:40758/0 (socket says 172.21.15.99:40758) 2024-03-25T18:31:42.242 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.241+0000 7facc3fff640 1 -- 172.21.15.99:0/1256405885 learned_addr learned my addr 172.21.15.99:0/1256405885 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:42.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.243+0000 7facabfff640 10 client.?.objecter ms_handle_connect 0x7facc41430c0 2024-03-25T18:31:42.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.243+0000 7facabfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:42.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.246+0000 7facabfff640 10 client.?.objecter ms_handle_connect 0x7facc413f710 2024-03-25T18:31:42.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.246+0000 7facabfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:42.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.254+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981579365 0 0) 0x7facc41db780 con 0x7facc41430c0 2024-03-25T18:31:42.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.254+0000 7facabfff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:42.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.254+0000 7facabfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:42.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.254+0000 7facabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:42.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.254+0000 7facabfff640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:31:42.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.254+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb0004eb0 con 0x7facc41430c0 2024-03-25T18:31:42.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.257+0000 7facabfff640 10 client.?.objecter ms_handle_connect 0x7facc40a98f0 2024-03-25T18:31:42.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.257+0000 7facabfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:42.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.257+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864479694 0 0) 0x7facc4139990 con 0x7facc413f710 2024-03-25T18:31:42.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.257+0000 7facabfff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:42.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.257+0000 7facabfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:42.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.257+0000 7facabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:42.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.257+0000 7facabfff640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:31:42.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.257+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc41db780 con 0x7facc413f710 2024-03-25T18:31:42.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.265+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303905203 0 0) 0x7facc40ab9c0 con 0x7facc40a98f0 2024-03-25T18:31:42.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.265+0000 7facabfff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:42.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.265+0000 7facabfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:42.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.265+0000 7facabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:42.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.265+0000 7facabfff640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:31:42.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.266+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc4139990 con 0x7facc40a98f0 2024-03-25T18:31:42.273 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.272+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1079660870 0 0) 0x7facb0004eb0 con 0x7facc41430c0 2024-03-25T18:31:42.273 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.272+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facc40ab9c0 con 0x7facc41430c0 2024-03-25T18:31:42.273 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.272+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7facbc001c90 con 0x7facc41430c0 2024-03-25T18:31:42.274 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.272+0000 7facabfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:42.274 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.272+0000 7facabfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:42.274 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.272+0000 7facabfff640 10 monclient(hunting): dump: 2024-03-25T18:31:42.274 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:42.274 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:42.274 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:42.274 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:42.274 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:42.274 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:42.274 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:42.274 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:42.274 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:42.274 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:42.274 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.272+0000 7facabfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:42.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.286+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1556038579 0 0) 0x7facc41db780 con 0x7facc413f710 2024-03-25T18:31:42.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.286+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facb0004eb0 con 0x7facc413f710 2024-03-25T18:31:42.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.286+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7facb8001570 con 0x7facc413f710 2024-03-25T18:31:42.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.286+0000 7facabfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:42.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.286+0000 7facabfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:42.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.286+0000 7facabfff640 10 monclient(hunting): dump: 2024-03-25T18:31:42.288 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:42.288 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:42.288 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:42.288 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:42.288 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:42.288 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:42.288 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:42.288 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:42.288 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:42.288 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:42.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.291+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3172086769 0 0) 0x7facc4139990 con 0x7facc40a98f0 2024-03-25T18:31:42.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.291+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facc41db780 con 0x7facc40a98f0 2024-03-25T18:31:42.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.291+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7facb40017b0 con 0x7facc40a98f0 2024-03-25T18:31:42.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.291+0000 7facabfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:42.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.291+0000 7facabfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:42.292 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.291+0000 7facabfff640 10 monclient(hunting): dump: 2024-03-25T18:31:42.292 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:42.292 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:42.293 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:42.293 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:42.293 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:42.293 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:42.293 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:42.293 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:42.293 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:42.293 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:42.301 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 141861098 0 0) 0x7facc40ab9c0 con 0x7facc41430c0 2024-03-25T18:31:42.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 >> v1:172.21.15.99:6790/0 conn(0x7facc413f710 legacy=0x7facc41db070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:42.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 >> v1:172.21.15.99:6789/0 conn(0x7facc40a98f0 legacy=0x7facc4138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:42.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facabfff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:42.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facabfff640 1 monclient: found mon.b 2024-03-25T18:31:42.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:42.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facabfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:42.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc4137ec0 con 0x7facc41430c0 2024-03-25T18:31:42.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facabfff640 10 monclient: _finish_auth 0 2024-03-25T18:31:42.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facabfff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:42.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:42.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facc9e05640 5 monclient: authenticate success, global_id 4513 2024-03-25T18:31:42.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facc9e05640 10 monclient: _renew_subs 2024-03-25T18:31:42.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facc9e05640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:42.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facc9e05640 1 -- 172.21.15.99:0/1256405885 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc4138170 con 0x7facc41430c0 2024-03-25T18:31:42.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facc9e05640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:42.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facc9e05640 10 monclient: _renew_subs 2024-03-25T18:31:42.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facc9e05640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:42.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facc9e05640 1 -- 172.21.15.99:0/1256405885 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facc4137ad0 con 0x7facc41430c0 2024-03-25T18:31:42.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.300+0000 7facc9e05640 1 librados: init done 2024-03-25T18:31:42.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.301+0000 7facc9e05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:42.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.301+0000 7facc9e05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:42.304 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.301+0000 7facc9e05640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:42.304 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.301+0000 7facc9e05640 1 -- 172.21.15.99:0/1256405885 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc40627b0 con 0x7facc41430c0 2024-03-25T18:31:42.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.592+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facbc003990 con 0x7facc41430c0 2024-03-25T18:31:42.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.592+0000 7facabfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:42.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.592+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7facbc003ce0 con 0x7facc41430c0 2024-03-25T18:31:42.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.592+0000 7facabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:42.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.592+0000 7facabfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:42.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.592+0000 7facabfff640 10 monclient: dump: 2024-03-25T18:31:42.594 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:42.594 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:42.594 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:42.595 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:42.595 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:42.595 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:42.595 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:42.595 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:42.595 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:42.595 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:42.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.592+0000 7facc1ffb640 4 set_mon_vals no callback set 2024-03-25T18:31:42.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.593+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7facbc003f00 con 0x7facc41430c0 2024-03-25T18:31:42.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.594+0000 7facc37fe640 10 monclient: get_auth_request con 0x7facb006e0f0 auth_method 0 2024-03-25T18:31:42.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.594+0000 7facabfff640 10 client.4513.objecter ms_handle_connect 0x7facb006e0f0 2024-03-25T18:31:42.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.598+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7facbc054ae0 con 0x7facc41430c0 2024-03-25T18:31:42.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.598+0000 7facabfff640 10 client.4513.objecter ms_dispatch 0x7facc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:42.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.598+0000 7facabfff640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:42.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.598+0000 7facabfff640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:42.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.598+0000 7facabfff640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-25T18:31:42.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.598+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facbc0505f0 con 0x7facc41430c0 2024-03-25T18:31:42.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.598+0000 7facabfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:42.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.598+0000 7facabfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:42.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.853+0000 7facc9e05640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:42.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.853+0000 7facc9e05640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:42.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.853+0000 7facc9e05640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:31:42.855 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.853+0000 7facc9e05640 1 -- 172.21.15.99:0/1256405885 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facc4141c00 con 0x7facc41430c0 2024-03-25T18:31:42.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.962+0000 7facabfff640 1 -- 172.21.15.99:0/1256405885 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7facbc052520 con 0x7facc41430c0 2024-03-25T18:31:42.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.962+0000 7facabfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:42.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.962+0000 7facabfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:42.964 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:31:42.964 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:42.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.965+0000 7facc9e05640 10 librados: watch_flush enter 2024-03-25T18:31:42.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.965+0000 7facc9e05640 10 librados: watch_flush exit 2024-03-25T18:31:42.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.965+0000 7facc9e05640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-25T18:31:42.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.965+0000 7facc9e05640 10 client.4513.objecter successfully canceled tick 2024-03-25T18:31:42.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.966+0000 7facc9e05640 1 -- 172.21.15.99:0/1256405885 >> v1:172.21.15.190:6800/4220409699 conn(0x7facb006e0f0 legacy=0x7facb00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:42.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.966+0000 7facc9e05640 10 monclient: shutdown 2024-03-25T18:31:42.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.966+0000 7facc9e05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:42.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.966+0000 7facc9e05640 1 -- 172.21.15.99:0/1256405885 >> v1:172.21.15.190:6789/0 conn(0x7facc41430c0 legacy=0x7facc41e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:42.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.966+0000 7facc9e05640 1 -- 172.21.15.99:0/1256405885 shutdown_connections 2024-03-25T18:31:42.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.966+0000 7facc9e05640 1 -- 172.21.15.99:0/1256405885 >> 172.21.15.99:0/1256405885 conn(0x7facc40b7ea0 msgr2=0x7facc4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:42.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.966+0000 7facc9e05640 1 -- 172.21.15.99:0/1256405885 shutdown_connections 2024-03-25T18:31:42.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.966+0000 7facc9e05640 1 -- 172.21.15.99:0/1256405885 wait complete. 2024-03-25T18:31:42.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:42.966+0000 7facc9e05640 1 librados: shutdown 2024-03-25T18:31:42.978 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:49.485 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:06:50.078 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:49.537 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:06:50.129 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:49.575 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:06:50.167 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:49.602 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:06:50.194 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:49.605 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:06:50.198 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:49.609 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:06:50.168 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:49.615 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:06:50.174 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:49.636 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:06:50.229 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:49.644 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:06:50.203 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:49.783 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:49.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.894+0000 7f235d9cd640 10 monclient: get_monmap_and_config 2024-03-25T18:31:49.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.894+0000 7f235d9cd640 10 monclient: build_initial_monmap 2024-03-25T18:31:49.898 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.894+0000 7f235d9cd640 10 monclient: monmap: 2024-03-25T18:31:49.898 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:49.898 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:49.898 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:31:49.895265+0000 2024-03-25T18:31:49.898 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:31:49.895265+0000 2024-03-25T18:31:49.898 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:49.898 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:49.898 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:49.898 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:31:49.898 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:31:49.898 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:49.899 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.894+0000 7f235d9cd640 1 Processor -- start 2024-03-25T18:31:49.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.894+0000 7f235d9cd640 1 -- start start 2024-03-25T18:31:49.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.894+0000 7f235d9cd640 10 monclient: init 2024-03-25T18:31:49.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.895+0000 7f235d9cd640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:49.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.895+0000 7f235d9cd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:31:49.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.895+0000 7f235d9cd640 10 monclient(hunting): picked mon.noname-b con 0x7f23580a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:49.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.895+0000 7f235d9cd640 10 monclient(hunting): picked mon.noname-a con 0x7f235813f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:49.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.895+0000 7f235d9cd640 10 monclient(hunting): picked mon.noname-c con 0x7f23581430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:49.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.895+0000 7f235d9cd640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2358142520 con 0x7f235813f710 2024-03-25T18:31:49.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.895+0000 7f235d9cd640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23580b28a0 con 0x7f23581430c0 2024-03-25T18:31:49.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.895+0000 7f235d9cd640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23580b2a30 con 0x7f23580a98f0 2024-03-25T18:31:49.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.895+0000 7f235d9cd640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:49.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.895+0000 7f235d9cd640 10 monclient(hunting): authenticate will time out at 7728.063524s 2024-03-25T18:31:49.910 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.909+0000 7f2356ffd640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f23580a98f0 0x7f23580a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:41636/0 (socket says 172.21.15.99:41636) 2024-03-25T18:31:49.910 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.909+0000 7f2356ffd640 1 -- 172.21.15.99:0/1397081284 learned_addr learned my addr 172.21.15.99:0/1397081284 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:49.919 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.917+0000 7f2355ffb640 1 -- 172.21.15.99:0/1397081284 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507059490 0 0) 0x7f23580b2a30 con 0x7f23580a98f0 2024-03-25T18:31:49.919 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.917+0000 7f2355ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:49.919 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.917+0000 7f2355ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:49.919 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.917+0000 7f2355ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:49.919 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.917+0000 7f2355ffb640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:31:49.919 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.917+0000 7f2355ffb640 1 -- 172.21.15.99:0/1397081284 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2340003520 con 0x7f23580a98f0 2024-03-25T18:31:49.920 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.919+0000 7f2355ffb640 1 -- 172.21.15.99:0/1397081284 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832343530 0 0) 0x7f23580b28a0 con 0x7f23581430c0 2024-03-25T18:31:49.920 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.919+0000 7f2355ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:49.920 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.919+0000 7f2355ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:49.920 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.919+0000 7f2355ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:49.920 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.919+0000 7f2355ffb640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:31:49.920 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.919+0000 7f2355ffb640 1 -- 172.21.15.99:0/1397081284 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23580b2a30 con 0x7f23581430c0 2024-03-25T18:31:49.932 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 1 -- 172.21.15.99:0/1397081284 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1510900180 0 0) 0x7f2340003520 con 0x7f23580a98f0 2024-03-25T18:31:49.932 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 1 -- 172.21.15.99:0/1397081284 >> v1:172.21.15.190:6789/0 conn(0x7f23581430c0 legacy=0x7f235814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:49.932 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 1 -- 172.21.15.99:0/1397081284 >> v1:172.21.15.99:6789/0 conn(0x7f235813f710 legacy=0x7f23581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:49.932 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:49.932 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 1 monclient: found mon.noname-b 2024-03-25T18:31:49.932 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:49.932 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:31:49.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 1 -- 172.21.15.99:0/1397081284 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23580b44c0 con 0x7f23580a98f0 2024-03-25T18:31:49.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:49.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:49.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:49.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 1 -- 172.21.15.99:0/1397081284 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2348001690 con 0x7f23580a98f0 2024-03-25T18:31:49.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:49.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:49.933 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f2355ffb640 10 monclient: dump: 2024-03-25T18:31:49.933 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:49.933 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:49.933 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:49.933 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:49.933 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:49.934 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:49.934 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:49.934 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:49.934 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:49.934 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:49.934 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f235d9cd640 5 monclient: authenticate success, global_id 4511 2024-03-25T18:31:49.934 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.930+0000 7f235d9cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:49.996 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.995+0000 7f2355ffb640 1 -- 172.21.15.99:0/1397081284 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23480017f0 con 0x7f23580a98f0 2024-03-25T18:31:49.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.995+0000 7f2355ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:49.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.995+0000 7f2355ffb640 1 -- 172.21.15.99:0/1397081284 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2348001c00 con 0x7f23580a98f0 2024-03-25T18:31:49.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.995+0000 7f235d9cd640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:49.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.995+0000 7f235d9cd640 4 set_mon_vals no callback set 2024-03-25T18:31:49.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.995+0000 7f2355ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:49.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.995+0000 7f2355ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:49.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.995+0000 7f2355ffb640 10 monclient: dump: 2024-03-25T18:31:49.997 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:49.997 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:49.997 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:49.997 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:49.997 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:49.997 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:49.997 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:49.998 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:49.998 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:49.998 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.996+0000 7f235d9cd640 10 monclient: shutdown 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.996+0000 7f235d9cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.996+0000 7f235d9cd640 1 -- 172.21.15.99:0/1397081284 >> v1:172.21.15.99:6790/0 conn(0x7f23580a98f0 legacy=0x7f23580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.996+0000 7f235d9cd640 1 -- 172.21.15.99:0/1397081284 shutdown_connections 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.996+0000 7f235d9cd640 1 -- 172.21.15.99:0/1397081284 >> 172.21.15.99:0/1397081284 conn(0x7f23580b7ea0 msgr2=0x7f23580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.996+0000 7f235d9cd640 1 -- 172.21.15.99:0/1397081284 shutdown_connections 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.996+0000 7f235d9cd640 1 -- 172.21.15.99:0/1397081284 wait complete. 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.997+0000 7f235d9cd640 10 monclient: build_initial_monmap 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.997+0000 7f235d9cd640 10 monclient: monmap: 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:50.000 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.997+0000 7f235d9cd640 1 librados: starting msgr at 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.997+0000 7f235d9cd640 1 librados: starting objecter 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.997+0000 7f235d9cd640 1 Processor -- start 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.997+0000 7f235d9cd640 1 -- start start 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.997+0000 7f235d9cd640 1 librados: setting wanted keys 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.997+0000 7f235d9cd640 1 librados: calling monclient init 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.997+0000 7f235d9cd640 10 monclient: init 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.998+0000 7f235d9cd640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.998+0000 7f235d9cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.998+0000 7f235d9cd640 10 monclient(hunting): picked mon.noname-c con 0x7f23580a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.998+0000 7f235d9cd640 10 monclient(hunting): picked mon.noname-b con 0x7f235813f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.998+0000 7f235d9cd640 10 monclient(hunting): picked mon.noname-a con 0x7f23581430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:50.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.998+0000 7f235d9cd640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23581db7d0 con 0x7f23581430c0 2024-03-25T18:31:50.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.998+0000 7f235d9cd640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23580b1420 con 0x7f235813f710 2024-03-25T18:31:50.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.998+0000 7f235d9cd640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23580b15b0 con 0x7f23580a98f0 2024-03-25T18:31:50.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.998+0000 7f235d9cd640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:50.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:49.998+0000 7f235d9cd640 10 monclient(hunting): authenticate will time out at 7728.166778s 2024-03-25T18:31:50.009 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.007+0000 7f2356ffd640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f23580a98f0 0x7f23581daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:41718/0 (socket says 172.21.15.99:41718) 2024-03-25T18:31:50.009 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.007+0000 7f2356ffd640 1 -- 172.21.15.99:0/945394178 learned_addr learned my addr 172.21.15.99:0/945394178 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:50.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.011+0000 7f233ffff640 10 client.?.objecter ms_handle_connect 0x7f235813f710 2024-03-25T18:31:50.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.011+0000 7f233ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:50.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.011+0000 7f233ffff640 10 client.?.objecter ms_handle_connect 0x7f23580a98f0 2024-03-25T18:31:50.012 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.011+0000 7f233ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:50.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.015+0000 7f233ffff640 10 client.?.objecter ms_handle_connect 0x7f23581430c0 2024-03-25T18:31:50.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.015+0000 7f233ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:50.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.019+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575388910 0 0) 0x7f23580b15b0 con 0x7f23580a98f0 2024-03-25T18:31:50.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.019+0000 7f233ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:50.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.019+0000 7f233ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:50.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.019+0000 7f233ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:50.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.019+0000 7f233ffff640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:31:50.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.019+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2344004eb0 con 0x7f23580a98f0 2024-03-25T18:31:50.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.025+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877887596 0 0) 0x7f23581db7d0 con 0x7f23581430c0 2024-03-25T18:31:50.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.025+0000 7f233ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:50.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.025+0000 7f233ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:50.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.025+0000 7f233ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:50.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.025+0000 7f233ffff640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:31:50.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.025+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23580b15b0 con 0x7f23581430c0 2024-03-25T18:31:50.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.026+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539994814 0 0) 0x7f23580b1420 con 0x7f235813f710 2024-03-25T18:31:50.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.026+0000 7f233ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:50.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.026+0000 7f233ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:50.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.026+0000 7f233ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:50.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.026+0000 7f233ffff640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:31:50.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.026+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23581db7d0 con 0x7f235813f710 2024-03-25T18:31:50.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.029+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1975532966 0 0) 0x7f2344004eb0 con 0x7f23580a98f0 2024-03-25T18:31:50.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.029+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23580b1420 con 0x7f23580a98f0 2024-03-25T18:31:50.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.029+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2348001c00 con 0x7f23580a98f0 2024-03-25T18:31:50.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.029+0000 7f233ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:50.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.029+0000 7f233ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:50.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.029+0000 7f233ffff640 10 monclient(hunting): dump: 2024-03-25T18:31:50.031 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:50.031 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:50.031 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:50.031 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:50.032 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:50.032 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:50.032 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:50.032 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:50.032 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:50.032 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:50.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.029+0000 7f233ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:50.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.033+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 550138667 0 0) 0x7f23580b15b0 con 0x7f23581430c0 2024-03-25T18:31:50.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.033+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2344004eb0 con 0x7f23581430c0 2024-03-25T18:31:50.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.033+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2350001b90 con 0x7f23581430c0 2024-03-25T18:31:50.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.033+0000 7f233ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:50.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.033+0000 7f233ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:50.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.033+0000 7f233ffff640 10 monclient(hunting): dump: 2024-03-25T18:31:50.035 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:50.035 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:50.035 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:50.035 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:50.035 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:50.036 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:50.036 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:50.036 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:50.036 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:50.036 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:50.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 410920258 0 0) 0x7f23580b1420 con 0x7f23580a98f0 2024-03-25T18:31:50.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 >> v1:172.21.15.190:6789/0 conn(0x7f235813f710 legacy=0x7f23580b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:50.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 >> v1:172.21.15.99:6789/0 conn(0x7f23581430c0 legacy=0x7f23581d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:50.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f233ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f233ffff640 1 monclient: found mon.c 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f233ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f233ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23580b1740 con 0x7f23580a98f0 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f233ffff640 10 monclient: _finish_auth 0 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f233ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f233ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f235d9cd640 5 monclient: authenticate success, global_id 4514 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f235d9cd640 10 monclient: _renew_subs 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f235d9cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f235d9cd640 1 -- 172.21.15.99:0/945394178 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23581db1a0 con 0x7f23580a98f0 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f235d9cd640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f235d9cd640 10 monclient: _renew_subs 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f235d9cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:31:50.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f235d9cd640 1 -- 172.21.15.99:0/945394178 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23580b08f0 con 0x7f23580a98f0 2024-03-25T18:31:50.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.037+0000 7f235d9cd640 1 librados: init done 2024-03-25T18:31:50.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.038+0000 7f235d9cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:50.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.038+0000 7f235d9cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:50.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.038+0000 7f235d9cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:31:50.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.038+0000 7f235d9cd640 1 -- 172.21.15.99:0/945394178 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23580627b0 con 0x7f23580a98f0 2024-03-25T18:31:50.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.339+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2348004a90 con 0x7f23580a98f0 2024-03-25T18:31:50.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.339+0000 7f233ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:50.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.339+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2348004e20 con 0x7f23580a98f0 2024-03-25T18:31:50.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.339+0000 7f233ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:50.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.339+0000 7f233ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:50.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.339+0000 7f233ffff640 10 monclient: dump: 2024-03-25T18:31:50.341 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:50.341 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:50.341 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:50.341 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:50.341 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:50.341 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:50.341 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:50.341 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:50.341 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:50.341 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:50.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.339+0000 7f23557fa640 4 set_mon_vals no callback set 2024-03-25T18:31:50.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.339+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f2348005040 con 0x7f23580a98f0 2024-03-25T18:31:50.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.340+0000 7f23567fc640 10 monclient: get_auth_request con 0x7f234406df70 auth_method 0 2024-03-25T18:31:50.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.340+0000 7f233ffff640 10 client.4514.objecter ms_handle_connect 0x7f234406df70 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.345+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f23480549b0 con 0x7f23580a98f0 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.345+0000 7f233ffff640 10 client.4514.objecter ms_dispatch 0x7f2358000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.345+0000 7f233ffff640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:50.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.345+0000 7f233ffff640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.345+0000 7f233ffff640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.345+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23480504c0 con 0x7f23580a98f0 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.345+0000 7f233ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:50.347 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.345+0000 7f233ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:50.581 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.579+0000 7f235d9cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:50.581 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.579+0000 7f235d9cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:50.581 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.579+0000 7f235d9cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:31:50.581 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.579+0000 7f235d9cd640 1 -- 172.21.15.99:0/945394178 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23580abb10 con 0x7f23580a98f0 2024-03-25T18:31:50.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.618+0000 7f233ffff640 1 -- 172.21.15.99:0/945394178 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f23480523f0 con 0x7f23580a98f0 2024-03-25T18:31:50.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.618+0000 7f233ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:50.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.618+0000 7f233ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:50.620 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:31:50.620 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:50.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.620+0000 7f235d9cd640 10 librados: watch_flush enter 2024-03-25T18:31:50.622 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.620+0000 7f235d9cd640 10 librados: watch_flush exit 2024-03-25T18:31:50.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.620+0000 7f235d9cd640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-25T18:31:50.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.620+0000 7f235d9cd640 10 client.4514.objecter successfully canceled tick 2024-03-25T18:31:50.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.620+0000 7f235d9cd640 1 -- 172.21.15.99:0/945394178 >> v1:172.21.15.190:6800/4220409699 conn(0x7f234406df70 legacy=0x7f2344070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:50.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.620+0000 7f235d9cd640 10 monclient: shutdown 2024-03-25T18:31:50.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.620+0000 7f235d9cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:50.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.620+0000 7f235d9cd640 1 -- 172.21.15.99:0/945394178 >> v1:172.21.15.99:6790/0 conn(0x7f23580a98f0 legacy=0x7f23581daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:50.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.621+0000 7f235d9cd640 1 -- 172.21.15.99:0/945394178 shutdown_connections 2024-03-25T18:31:50.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.621+0000 7f235d9cd640 1 -- 172.21.15.99:0/945394178 >> 172.21.15.99:0/945394178 conn(0x7f23580b7ea0 msgr2=0x7f235813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:50.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.621+0000 7f235d9cd640 1 -- 172.21.15.99:0/945394178 shutdown_connections 2024-03-25T18:31:50.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.621+0000 7f235d9cd640 1 -- 172.21.15.99:0/945394178 wait complete. 2024-03-25T18:31:50.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:50.621+0000 7f235d9cd640 1 librados: shutdown 2024-03-25T18:31:50.634 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:57.441 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:57.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.553+0000 7f71be782640 10 monclient: get_monmap_and_config 2024-03-25T18:31:57.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.553+0000 7f71be782640 10 monclient: build_initial_monmap 2024-03-25T18:31:57.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.553+0000 7f71be782640 10 monclient: monmap: 2024-03-25T18:31:57.555 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:57.555 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:57.555 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:31:57.554617+0000 2024-03-25T18:31:57.555 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:31:57.554617+0000 2024-03-25T18:31:57.555 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:57.555 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:57.555 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:57.555 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:31:57.555 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:31:57.555 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:57.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.554+0000 7f71be782640 1 Processor -- start 2024-03-25T18:31:57.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.554+0000 7f71be782640 1 -- start start 2024-03-25T18:31:57.557 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.554+0000 7f71be782640 10 monclient: init 2024-03-25T18:31:57.557 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.554+0000 7f71be782640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:57.557 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.554+0000 7f71be782640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:57.557 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.555+0000 7f71be782640 10 monclient(hunting): picked mon.noname-a con 0x7f71b80a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:57.557 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.555+0000 7f71be782640 10 monclient(hunting): picked mon.noname-b con 0x7f71b813f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:57.557 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.555+0000 7f71be782640 10 monclient(hunting): picked mon.noname-c con 0x7f71b81430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:57.557 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.555+0000 7f71be782640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b8142520 con 0x7f71b80a98f0 2024-03-25T18:31:57.557 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.555+0000 7f71be782640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b80b28a0 con 0x7f71b81430c0 2024-03-25T18:31:57.557 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.555+0000 7f71be782640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b80b2a30 con 0x7f71b813f710 2024-03-25T18:31:57.557 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.555+0000 7f71be782640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:57.557 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.555+0000 7f71be782640 10 monclient(hunting): authenticate will time out at 7735.722961s 2024-03-25T18:31:57.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.565+0000 7f71b7fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f71b80a98f0 0x7f71b80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56606/0 (socket says 172.21.15.99:56606) 2024-03-25T18:31:57.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.565+0000 7f71b7fff640 1 -- 172.21.15.99:0/3015381623 learned_addr learned my addr 172.21.15.99:0/3015381623 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:57.573 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.572+0000 7f71b6ffd640 1 -- 172.21.15.99:0/3015381623 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599806978 0 0) 0x7f71b8142520 con 0x7f71b80a98f0 2024-03-25T18:31:57.573 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.572+0000 7f71b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:57.573 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.572+0000 7f71b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:57.573 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.572+0000 7f71b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:57.574 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.572+0000 7f71b6ffd640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:31:57.574 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.572+0000 7f71b6ffd640 1 -- 172.21.15.99:0/3015381623 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71a0003520 con 0x7f71b80a98f0 2024-03-25T18:31:57.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 1 -- 172.21.15.99:0/3015381623 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3993197040 0 0) 0x7f71a0003520 con 0x7f71b80a98f0 2024-03-25T18:31:57.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 1 -- 172.21.15.99:0/3015381623 >> v1:172.21.15.99:6790/0 conn(0x7f71b813f710 legacy=0x7f71b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:57.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 1 -- 172.21.15.99:0/3015381623 >> v1:172.21.15.190:6789/0 conn(0x7f71b81430c0 legacy=0x7f71b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:57.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:57.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 1 monclient: found mon.noname-a 2024-03-25T18:31:57.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:57.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:31:57.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 1 -- 172.21.15.99:0/3015381623 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b80b44c0 con 0x7f71b80a98f0 2024-03-25T18:31:57.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 1 -- 172.21.15.99:0/3015381623 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f71a8001690 con 0x7f71b80a98f0 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.576+0000 7f71b6ffd640 10 monclient: dump: 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:57.579 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:57.580 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:57.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.577+0000 7f71be782640 5 monclient: authenticate success, global_id 4506 2024-03-25T18:31:57.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.577+0000 7f71be782640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:57.587 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.586+0000 7f71b6ffd640 1 -- 172.21.15.99:0/3015381623 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71a80017f0 con 0x7f71b80a98f0 2024-03-25T18:31:57.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.586+0000 7f71b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:57.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.586+0000 7f71b6ffd640 1 -- 172.21.15.99:0/3015381623 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f71a8001c40 con 0x7f71b80a98f0 2024-03-25T18:31:57.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.586+0000 7f71b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:57.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.586+0000 7f71b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:57.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.586+0000 7f71b6ffd640 10 monclient: dump: 2024-03-25T18:31:57.588 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:57.588 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:57.588 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:57.588 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:57.588 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:57.588 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:57.588 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:57.588 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:57.588 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:57.589 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:57.589 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.586+0000 7f71be782640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:57.589 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.586+0000 7f71be782640 4 set_mon_vals no callback set 2024-03-25T18:31:57.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.587+0000 7f71be782640 10 monclient: shutdown 2024-03-25T18:31:57.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.587+0000 7f71be782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:57.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.587+0000 7f71be782640 1 -- 172.21.15.99:0/3015381623 >> v1:172.21.15.99:6789/0 conn(0x7f71b80a98f0 legacy=0x7f71b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:57.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.587+0000 7f71be782640 1 -- 172.21.15.99:0/3015381623 shutdown_connections 2024-03-25T18:31:57.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.587+0000 7f71be782640 1 -- 172.21.15.99:0/3015381623 >> 172.21.15.99:0/3015381623 conn(0x7f71b80b7ea0 msgr2=0x7f71b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:57.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.587+0000 7f71be782640 1 -- 172.21.15.99:0/3015381623 shutdown_connections 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.587+0000 7f71be782640 1 -- 172.21.15.99:0/3015381623 wait complete. 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.588+0000 7f71be782640 10 monclient: build_initial_monmap 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.588+0000 7f71be782640 10 monclient: monmap: 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.589+0000 7f71be782640 1 librados: starting msgr at 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.589+0000 7f71be782640 1 librados: starting objecter 2024-03-25T18:31:57.591 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.589+0000 7f71be782640 1 Processor -- start 2024-03-25T18:31:57.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.589+0000 7f71be782640 1 -- start start 2024-03-25T18:31:57.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.589+0000 7f71be782640 1 librados: setting wanted keys 2024-03-25T18:31:57.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.589+0000 7f71be782640 1 librados: calling monclient init 2024-03-25T18:31:57.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.589+0000 7f71be782640 10 monclient: init 2024-03-25T18:31:57.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.590+0000 7f71be782640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:57.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.590+0000 7f71be782640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:57.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.590+0000 7f71be782640 10 monclient(hunting): picked mon.noname-a con 0x7f71b80a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:31:57.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.590+0000 7f71be782640 10 monclient(hunting): picked mon.noname-b con 0x7f71b813f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:31:57.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.590+0000 7f71be782640 10 monclient(hunting): picked mon.noname-c con 0x7f71b81430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:31:57.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.590+0000 7f71be782640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b80b7990 con 0x7f71b80a98f0 2024-03-25T18:31:57.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.590+0000 7f71be782640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b8137080 con 0x7f71b813f710 2024-03-25T18:31:57.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.590+0000 7f71be782640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b81351c0 con 0x7f71b81430c0 2024-03-25T18:31:57.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.590+0000 7f71be782640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:57.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.590+0000 7f71be782640 10 monclient(hunting): authenticate will time out at 7735.758335s 2024-03-25T18:31:57.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.598+0000 7f71b7fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f71b80a98f0 0x7f71b81daa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56648/0 (socket says 172.21.15.99:56648) 2024-03-25T18:31:57.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.598+0000 7f71b7fff640 1 -- 172.21.15.99:0/2741663545 learned_addr learned my addr 172.21.15.99:0/2741663545 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:31:57.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.600+0000 7f71b4ff9640 10 client.?.objecter ms_handle_connect 0x7f71b80a98f0 2024-03-25T18:31:57.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.600+0000 7f71b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:57.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.612+0000 7f71b4ff9640 10 client.?.objecter ms_handle_connect 0x7f71b81430c0 2024-03-25T18:31:57.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.612+0000 7f71b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:57.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.613+0000 7f71b4ff9640 10 client.?.objecter ms_handle_connect 0x7f71b813f710 2024-03-25T18:31:57.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.613+0000 7f71b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:57.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.617+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93878273 0 0) 0x7f71b81351c0 con 0x7f71b81430c0 2024-03-25T18:31:57.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.617+0000 7f71b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:57.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.617+0000 7f71b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:57.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.617+0000 7f71b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:57.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.617+0000 7f71b4ff9640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:31:57.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.617+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71a4004eb0 con 0x7f71b81430c0 2024-03-25T18:31:57.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.625+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699960135 0 0) 0x7f71b8137080 con 0x7f71b813f710 2024-03-25T18:31:57.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.625+0000 7f71b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:57.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.625+0000 7f71b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:57.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.625+0000 7f71b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:57.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.625+0000 7f71b4ff9640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:31:57.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.625+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b81351c0 con 0x7f71b813f710 2024-03-25T18:31:57.628 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.627+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031860531 0 0) 0x7f71b80b7990 con 0x7f71b80a98f0 2024-03-25T18:31:57.629 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.627+0000 7f71b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:57.629 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.627+0000 7f71b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:57.629 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.627+0000 7f71b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:57.629 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.627+0000 7f71b4ff9640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:31:57.629 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.627+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b8137080 con 0x7f71b80a98f0 2024-03-25T18:31:57.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.630+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3391631805 0 0) 0x7f71a4004eb0 con 0x7f71b81430c0 2024-03-25T18:31:57.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.630+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71b80b7990 con 0x7f71b81430c0 2024-03-25T18:31:57.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.630+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f71b0001b20 con 0x7f71b81430c0 2024-03-25T18:31:57.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.630+0000 7f71b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:57.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.630+0000 7f71b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:57.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.630+0000 7f71b4ff9640 10 monclient(hunting): dump: 2024-03-25T18:31:57.632 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:57.632 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:57.632 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:57.632 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:57.632 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:57.632 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:57.632 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:57.632 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:57.632 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:57.632 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:57.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.630+0000 7f71b4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:57.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.641+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 199310477 0 0) 0x7f71b81351c0 con 0x7f71b813f710 2024-03-25T18:31:57.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.641+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71a4004eb0 con 0x7f71b813f710 2024-03-25T18:31:57.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.641+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f71ac0018b0 con 0x7f71b813f710 2024-03-25T18:31:57.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.641+0000 7f71b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:57.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.641+0000 7f71b4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:57.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.641+0000 7f71b4ff9640 10 monclient(hunting): dump: 2024-03-25T18:31:57.643 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:57.643 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:57.643 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:57.643 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:57.643 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:57.643 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:57.643 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:57.643 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:57.643 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:57.643 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:57.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.643+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1652601553 0 0) 0x7f71b80b7990 con 0x7f71b81430c0 2024-03-25T18:31:57.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.643+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 >> v1:172.21.15.190:6789/0 conn(0x7f71b813f710 legacy=0x7f71b81db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:57.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.643+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 >> v1:172.21.15.99:6789/0 conn(0x7f71b80a98f0 legacy=0x7f71b81daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:57.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.643+0000 7f71b4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:31:57.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.643+0000 7f71b4ff9640 1 monclient: found mon.c 2024-03-25T18:31:57.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.643+0000 7f71b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:57.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.643+0000 7f71b4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:31:57.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.643+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b81354e0 con 0x7f71b81430c0 2024-03-25T18:31:57.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.643+0000 7f71b4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:31:57.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.643+0000 7f71b4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:31:57.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.643+0000 7f71b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:57.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.644+0000 7f71be782640 5 monclient: authenticate success, global_id 4520 2024-03-25T18:31:57.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.644+0000 7f71be782640 10 monclient: _renew_subs 2024-03-25T18:31:57.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.644+0000 7f71be782640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:31:57.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.644+0000 7f71be782640 1 -- 172.21.15.99:0/2741663545 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b81357b0 con 0x7f71b81430c0 2024-03-25T18:31:57.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.644+0000 7f71be782640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:57.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.644+0000 7f71be782640 10 monclient: _renew_subs 2024-03-25T18:31:57.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.644+0000 7f71be782640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:31:57.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.644+0000 7f71be782640 1 -- 172.21.15.99:0/2741663545 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b813b7f0 con 0x7f71b81430c0 2024-03-25T18:31:57.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.644+0000 7f71be782640 1 librados: init done 2024-03-25T18:31:57.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.645+0000 7f71be782640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:57.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.645+0000 7f71be782640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:57.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.645+0000 7f71be782640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:31:57.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.645+0000 7f71be782640 1 -- 172.21.15.99:0/2741663545 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b80b76f0 con 0x7f71b81430c0 2024-03-25T18:31:57.949 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.947+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71b0002030 con 0x7f71b81430c0 2024-03-25T18:31:57.949 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.947+0000 7f71b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:57.949 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.947+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f71b0003b10 con 0x7f71b81430c0 2024-03-25T18:31:57.949 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.947+0000 7f71b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:57.949 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.947+0000 7f71b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:57.949 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.947+0000 7f71b67fc640 4 set_mon_vals no callback set 2024-03-25T18:31:57.949 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.947+0000 7f71b4ff9640 10 monclient: dump: 2024-03-25T18:31:57.949 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:31:57.949 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:31:57.949 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:57.949 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:31:57.950 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:57.950 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:31:57.950 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:31:57.950 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:31:57.950 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:31:57.950 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:31:57.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.948+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f71b0019ae0 con 0x7f71b81430c0 2024-03-25T18:31:57.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.949+0000 7f71b7fff640 10 monclient: get_auth_request con 0x7f71a406df70 auth_method 0 2024-03-25T18:31:57.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.949+0000 7f71b4ff9640 10 client.4520.objecter ms_handle_connect 0x7f71a406df70 2024-03-25T18:31:57.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.954+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f71b0053760 con 0x7f71b81430c0 2024-03-25T18:31:57.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.954+0000 7f71b4ff9640 10 client.4520.objecter ms_dispatch 0x7f71b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:57.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.954+0000 7f71b4ff9640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:57.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.954+0000 7f71b4ff9640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:57.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.954+0000 7f71b4ff9640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-25T18:31:57.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.954+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f71b004f0a0 con 0x7f71b81430c0 2024-03-25T18:31:57.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.954+0000 7f71b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:57.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:57.954+0000 7f71b4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:58.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.190+0000 7f71be782640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:58.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.190+0000 7f71be782640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:58.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.190+0000 7f71be782640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:31:58.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.190+0000 7f71be782640 1 -- 172.21.15.99:0/2741663545 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71b80b7b40 con 0x7f71b81430c0 2024-03-25T18:31:58.240 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.239+0000 7f71b4ff9640 1 -- 172.21.15.99:0/2741663545 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f71b80b7b40 con 0x7f71b81430c0 2024-03-25T18:31:58.240 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.239+0000 7f71b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:58.240 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.239+0000 7f71b4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:58.240 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:31:58.240 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:58.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.241+0000 7f71be782640 10 librados: watch_flush enter 2024-03-25T18:31:58.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.241+0000 7f71be782640 10 librados: watch_flush exit 2024-03-25T18:31:58.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.241+0000 7f71be782640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-25T18:31:58.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.241+0000 7f71be782640 10 client.4520.objecter successfully canceled tick 2024-03-25T18:31:58.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.241+0000 7f71be782640 1 -- 172.21.15.99:0/2741663545 >> v1:172.21.15.190:6800/4220409699 conn(0x7f71a406df70 legacy=0x7f71a4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:58.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.241+0000 7f71be782640 10 monclient: shutdown 2024-03-25T18:31:58.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.241+0000 7f71be782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:58.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.241+0000 7f71be782640 1 -- 172.21.15.99:0/2741663545 >> v1:172.21.15.99:6790/0 conn(0x7f71b81430c0 legacy=0x7f71b81e1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:58.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.241+0000 7f71be782640 1 -- 172.21.15.99:0/2741663545 shutdown_connections 2024-03-25T18:31:58.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.241+0000 7f71be782640 1 -- 172.21.15.99:0/2741663545 >> 172.21.15.99:0/2741663545 conn(0x7f71b80b7ea0 msgr2=0x7f71b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:58.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.242+0000 7f71be782640 1 -- 172.21.15.99:0/2741663545 shutdown_connections 2024-03-25T18:31:58.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.242+0000 7f71be782640 1 -- 172.21.15.99:0/2741663545 wait complete. 2024-03-25T18:31:58.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:31:58.242+0000 7f71be782640 1 librados: shutdown 2024-03-25T18:31:58.254 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:05.061 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:05.174 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.173+0000 7f835f1b5640 10 monclient: get_monmap_and_config 2024-03-25T18:32:05.175 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.173+0000 7f835f1b5640 10 monclient: build_initial_monmap 2024-03-25T18:32:05.175 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.173+0000 7f835f1b5640 10 monclient: monmap: 2024-03-25T18:32:05.175 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:05.175 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:05.175 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:32:05.174644+0000 2024-03-25T18:32:05.175 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:32:05.174644+0000 2024-03-25T18:32:05.175 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:05.175 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:05.175 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:05.175 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:32:05.175 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:32:05.175 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:05.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.174+0000 7f835f1b5640 1 Processor -- start 2024-03-25T18:32:05.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.174+0000 7f835f1b5640 1 -- start start 2024-03-25T18:32:05.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.174+0000 7f835f1b5640 10 monclient: init 2024-03-25T18:32:05.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.175+0000 7f835f1b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:05.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.175+0000 7f835f1b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:05.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.175+0000 7f835f1b5640 10 monclient(hunting): picked mon.noname-a con 0x7f83580a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:05.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.175+0000 7f835f1b5640 10 monclient(hunting): picked mon.noname-b con 0x7f835813f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:05.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.175+0000 7f835f1b5640 10 monclient(hunting): picked mon.noname-c con 0x7f83581430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:05.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.175+0000 7f835f1b5640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8358142520 con 0x7f83580a98f0 2024-03-25T18:32:05.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.175+0000 7f835f1b5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83580b28a0 con 0x7f83581430c0 2024-03-25T18:32:05.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.175+0000 7f835f1b5640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83580b2a30 con 0x7f835813f710 2024-03-25T18:32:05.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.175+0000 7f835f1b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:05.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.175+0000 7f835f1b5640 10 monclient(hunting): authenticate will time out at 7743.343228s 2024-03-25T18:32:05.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.182+0000 7f835cf51640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f83580a98f0 0x7f83580a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:42996/0 (socket says 172.21.15.99:42996) 2024-03-25T18:32:05.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.182+0000 7f835cf51640 1 -- 172.21.15.99:0/4224944761 learned_addr learned my addr 172.21.15.99:0/4224944761 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:05.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.198+0000 7f83577fe640 1 -- 172.21.15.99:0/4224944761 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862781182 0 0) 0x7f8358142520 con 0x7f83580a98f0 2024-03-25T18:32:05.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.198+0000 7f83577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:05.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.198+0000 7f83577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:05.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.198+0000 7f83577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:05.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.198+0000 7f83577fe640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:32:05.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.198+0000 7f83577fe640 1 -- 172.21.15.99:0/4224944761 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8348003520 con 0x7f83580a98f0 2024-03-25T18:32:05.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.198+0000 7f83577fe640 1 -- 172.21.15.99:0/4224944761 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682343411 0 0) 0x7f83580b28a0 con 0x7f83581430c0 2024-03-25T18:32:05.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.198+0000 7f83577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:05.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.198+0000 7f83577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:05.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.198+0000 7f83577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:05.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.198+0000 7f83577fe640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:32:05.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.198+0000 7f83577fe640 1 -- 172.21.15.99:0/4224944761 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8358142520 con 0x7f83581430c0 2024-03-25T18:32:05.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.212+0000 7f83577fe640 1 -- 172.21.15.99:0/4224944761 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129633291 0 0) 0x7f83580b2a30 con 0x7f835813f710 2024-03-25T18:32:05.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.212+0000 7f83577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:05.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.212+0000 7f83577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:05.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.212+0000 7f83577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:05.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.212+0000 7f83577fe640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:32:05.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.212+0000 7f83577fe640 1 -- 172.21.15.99:0/4224944761 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83580b28a0 con 0x7f835813f710 2024-03-25T18:32:05.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 1 -- 172.21.15.99:0/4224944761 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3246959148 0 0) 0x7f8348003520 con 0x7f83580a98f0 2024-03-25T18:32:05.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 1 -- 172.21.15.99:0/4224944761 >> v1:172.21.15.99:6790/0 conn(0x7f835813f710 legacy=0x7f83581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 1 -- 172.21.15.99:0/4224944761 >> v1:172.21.15.190:6789/0 conn(0x7f83581430c0 legacy=0x7f835814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 1 monclient: found mon.noname-a 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 1 -- 172.21.15.99:0/4224944761 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83580b44c0 con 0x7f83580a98f0 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 1 -- 172.21.15.99:0/4224944761 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f834c001690 con 0x7f83580a98f0 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f83577fe640 10 monclient: dump: 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:05.216 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:05.217 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:05.217 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:05.217 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:05.217 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:05.217 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:05.217 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f835f1b5640 5 monclient: authenticate success, global_id 4512 2024-03-25T18:32:05.217 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.214+0000 7f835f1b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:05.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.227+0000 7f83577fe640 1 -- 172.21.15.99:0/4224944761 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f834c0017f0 con 0x7f83580a98f0 2024-03-25T18:32:05.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.227+0000 7f83577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:05.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.227+0000 7f83577fe640 1 -- 172.21.15.99:0/4224944761 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f834c001c00 con 0x7f83580a98f0 2024-03-25T18:32:05.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.227+0000 7f835f1b5640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:05.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.227+0000 7f835f1b5640 4 set_mon_vals no callback set 2024-03-25T18:32:05.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.227+0000 7f83577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:05.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.227+0000 7f83577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:05.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.227+0000 7f83577fe640 10 monclient: dump: 2024-03-25T18:32:05.228 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:05.228 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:05.229 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:05.229 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:05.229 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:05.229 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:05.229 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:05.229 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:05.229 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:05.229 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:05.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.227+0000 7f835f1b5640 10 monclient: shutdown 2024-03-25T18:32:05.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.228+0000 7f835f1b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:05.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.228+0000 7f835f1b5640 1 -- 172.21.15.99:0/4224944761 >> v1:172.21.15.99:6789/0 conn(0x7f83580a98f0 legacy=0x7f83580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.228+0000 7f835f1b5640 1 -- 172.21.15.99:0/4224944761 shutdown_connections 2024-03-25T18:32:05.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.228+0000 7f835f1b5640 1 -- 172.21.15.99:0/4224944761 >> 172.21.15.99:0/4224944761 conn(0x7f83580b7ea0 msgr2=0x7f83580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:05.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.228+0000 7f835f1b5640 1 -- 172.21.15.99:0/4224944761 shutdown_connections 2024-03-25T18:32:05.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.228+0000 7f835f1b5640 1 -- 172.21.15.99:0/4224944761 wait complete. 2024-03-25T18:32:05.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.229+0000 7f835f1b5640 10 monclient: build_initial_monmap 2024-03-25T18:32:05.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.229+0000 7f835f1b5640 10 monclient: monmap: 2024-03-25T18:32:05.231 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:05.231 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:05.231 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:32:05.231 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:32:05.231 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:05.232 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:32:05.232 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:05.232 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:32:05.232 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:32:05.232 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:05.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.230+0000 7f835f1b5640 1 librados: starting msgr at 2024-03-25T18:32:05.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.230+0000 7f835f1b5640 1 librados: starting objecter 2024-03-25T18:32:05.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.230+0000 7f835f1b5640 1 Processor -- start 2024-03-25T18:32:05.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.230+0000 7f835f1b5640 1 -- start start 2024-03-25T18:32:05.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.230+0000 7f835f1b5640 1 librados: setting wanted keys 2024-03-25T18:32:05.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.230+0000 7f835f1b5640 1 librados: calling monclient init 2024-03-25T18:32:05.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.230+0000 7f835f1b5640 10 monclient: init 2024-03-25T18:32:05.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.231+0000 7f835f1b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:05.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.231+0000 7f835f1b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:05.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.231+0000 7f835f1b5640 10 monclient(hunting): picked mon.noname-c con 0x7f835813f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:05.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.231+0000 7f835f1b5640 10 monclient(hunting): picked mon.noname-b con 0x7f83581430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:05.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.231+0000 7f835f1b5640 10 monclient(hunting): picked mon.noname-a con 0x7f835813b9a0 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:05.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.231+0000 7f835f1b5640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8358139520 con 0x7f835813b9a0 2024-03-25T18:32:05.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.231+0000 7f835f1b5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8358138b40 con 0x7f83581430c0 2024-03-25T18:32:05.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.231+0000 7f835f1b5640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8358138e20 con 0x7f835813f710 2024-03-25T18:32:05.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.231+0000 7f835f1b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:05.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.231+0000 7f835f1b5640 10 monclient(hunting): authenticate will time out at 7743.399488s 2024-03-25T18:32:05.246 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.245+0000 7f8357fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f83581430c0 0x7f8358137b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:36384/0 (socket says 172.21.15.99:36384) 2024-03-25T18:32:05.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.245+0000 7f8357fff640 1 -- 172.21.15.99:0/123572280 learned_addr learned my addr 172.21.15.99:0/123572280 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:05.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.246+0000 7f83557fa640 10 client.?.objecter ms_handle_connect 0x7f83581430c0 2024-03-25T18:32:05.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.246+0000 7f83557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:05.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.253+0000 7f83557fa640 10 client.?.objecter ms_handle_connect 0x7f835813f710 2024-03-25T18:32:05.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.253+0000 7f83557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:05.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.257+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965079948 0 0) 0x7f8358138b40 con 0x7f83581430c0 2024-03-25T18:32:05.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.257+0000 7f83557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:05.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.257+0000 7f83557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:05.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.257+0000 7f83557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:05.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.257+0000 7f83557fa640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:32:05.258 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.257+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f833c004eb0 con 0x7f83581430c0 2024-03-25T18:32:05.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.266+0000 7f83557fa640 10 client.?.objecter ms_handle_connect 0x7f835813b9a0 2024-03-25T18:32:05.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.266+0000 7f83557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.277+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3444887046 0 0) 0x7f833c004eb0 con 0x7f83581430c0 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.277+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8358138b40 con 0x7f83581430c0 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.277+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8344001a10 con 0x7f83581430c0 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.277+0000 7f83557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.277+0000 7f83557fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.277+0000 7f83557fa640 10 monclient(hunting): dump: 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:05.279 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:05.280 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:05.280 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.277+0000 7f83557fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:05.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.282+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956707676 0 0) 0x7f8358139520 con 0x7f835813b9a0 2024-03-25T18:32:05.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.282+0000 7f83557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:05.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.282+0000 7f83557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:05.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.282+0000 7f83557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:05.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.282+0000 7f83557fa640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:32:05.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.282+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f833c004eb0 con 0x7f835813b9a0 2024-03-25T18:32:05.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.283+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076713563 0 0) 0x7f8358138e20 con 0x7f835813f710 2024-03-25T18:32:05.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.283+0000 7f83557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:05.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.283+0000 7f83557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:05.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.283+0000 7f83557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:05.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.283+0000 7f83557fa640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:32:05.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.283+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8358139520 con 0x7f835813f710 2024-03-25T18:32:05.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4045095931 0 0) 0x7f8358138b40 con 0x7f83581430c0 2024-03-25T18:32:05.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 >> v1:172.21.15.99:6790/0 conn(0x7f835813f710 legacy=0x7f83581373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 >> v1:172.21.15.99:6789/0 conn(0x7f835813b9a0 legacy=0x7f83581e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f83557fa640 10 monclient: _finish_hunting 0 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f83557fa640 1 monclient: found mon.b 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f83557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f83557fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8358139100 con 0x7f83581430c0 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f83557fa640 10 monclient: _finish_auth 0 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f83557fa640 10 monclient: _check_auth_tickets 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f83557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f835f1b5640 5 monclient: authenticate success, global_id 4531 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f835f1b5640 10 monclient: _renew_subs 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f835f1b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f835f1b5640 1 -- 172.21.15.99:0/123572280 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8358139750 con 0x7f83581430c0 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f835f1b5640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:05.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f835f1b5640 10 monclient: _renew_subs 2024-03-25T18:32:05.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f835f1b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:05.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f835f1b5640 1 -- 172.21.15.99:0/123572280 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8358136c10 con 0x7f83581430c0 2024-03-25T18:32:05.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.293+0000 7f835f1b5640 1 librados: init done 2024-03-25T18:32:05.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.294+0000 7f835f1b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:05.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.294+0000 7f835f1b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:05.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.294+0000 7f835f1b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:05.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.294+0000 7f835f1b5640 1 -- 172.21.15.99:0/123572280 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83580b3ef0 con 0x7f83581430c0 2024-03-25T18:32:05.576 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.575+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8344001cb0 con 0x7f83581430c0 2024-03-25T18:32:05.576 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.575+0000 7f83557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:05.576 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.575+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f8344003a60 con 0x7f83581430c0 2024-03-25T18:32:05.576 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.575+0000 7f83557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:05.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.575+0000 7f83557fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:05.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.575+0000 7f83557fa640 10 monclient: dump: 2024-03-25T18:32:05.577 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:05.577 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:05.577 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:05.577 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:05.577 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:05.577 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:05.577 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:05.577 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:05.577 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:05.577 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:05.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.575+0000 7f8356ffd640 4 set_mon_vals no callback set 2024-03-25T18:32:05.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.576+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f8344003c40 con 0x7f83581430c0 2024-03-25T18:32:05.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.576+0000 7f835cf51640 10 monclient: get_auth_request con 0x7f833c06dea0 auth_method 0 2024-03-25T18:32:05.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.577+0000 7f83557fa640 10 client.4531.objecter ms_handle_connect 0x7f833c06dea0 2024-03-25T18:32:05.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.582+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f83440548f0 con 0x7f83581430c0 2024-03-25T18:32:05.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.582+0000 7f83557fa640 10 client.4531.objecter ms_dispatch 0x7f8358000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:05.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.582+0000 7f83557fa640 3 client.4531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:05.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.582+0000 7f83557fa640 3 client.4531.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:05.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.582+0000 7f83557fa640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-25T18:32:05.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.582+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f83440b18b0 con 0x7f83581430c0 2024-03-25T18:32:05.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.582+0000 7f83557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:05.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.582+0000 7f83557fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:05.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.829+0000 7f835f1b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:05.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.829+0000 7f835f1b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:05.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.829+0000 7f835f1b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:05.830 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.829+0000 7f835f1b5640 1 -- 172.21.15.99:0/123572280 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83580627b0 con 0x7f83581430c0 2024-03-25T18:32:05.861 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.860+0000 7f83557fa640 1 -- 172.21.15.99:0/123572280 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f8344053fc0 con 0x7f83581430c0 2024-03-25T18:32:05.861 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.860+0000 7f83557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:05.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.860+0000 7f83557fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:05.862 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:32:05.862 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:05.864 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.863+0000 7f835f1b5640 10 librados: watch_flush enter 2024-03-25T18:32:05.864 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.863+0000 7f835f1b5640 10 librados: watch_flush exit 2024-03-25T18:32:05.864 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.863+0000 7f835f1b5640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-25T18:32:05.865 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.863+0000 7f835f1b5640 10 client.4531.objecter successfully canceled tick 2024-03-25T18:32:05.865 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.863+0000 7f835f1b5640 1 -- 172.21.15.99:0/123572280 >> v1:172.21.15.190:6800/4220409699 conn(0x7f833c06dea0 legacy=0x7f833c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.865 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.863+0000 7f835f1b5640 10 monclient: shutdown 2024-03-25T18:32:05.865 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.863+0000 7f835f1b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:05.865 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.863+0000 7f835f1b5640 1 -- 172.21.15.99:0/123572280 >> v1:172.21.15.190:6789/0 conn(0x7f83581430c0 legacy=0x7f8358137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:05.865 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.863+0000 7f835f1b5640 1 -- 172.21.15.99:0/123572280 shutdown_connections 2024-03-25T18:32:05.865 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.863+0000 7f835f1b5640 1 -- 172.21.15.99:0/123572280 >> 172.21.15.99:0/123572280 conn(0x7f83580b7ea0 msgr2=0x7f83580b7630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:05.865 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.863+0000 7f835f1b5640 1 -- 172.21.15.99:0/123572280 shutdown_connections 2024-03-25T18:32:05.865 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.863+0000 7f835f1b5640 1 -- 172.21.15.99:0/123572280 wait complete. 2024-03-25T18:32:05.865 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:05.863+0000 7f835f1b5640 1 librados: shutdown 2024-03-25T18:32:05.875 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:09.553 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:07:10.145 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:09.606 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:07:10.198 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:09.645 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:07:10.237 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:09.655 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:07:10.214 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:09.665 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:07:10.224 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:09.668 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:07:10.261 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:09.676 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:07:10.269 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:09.711 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:07:10.303 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:09.713 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:07:10.272 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:12.681 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:12.796 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.794+0000 7fb64b47b640 10 monclient: get_monmap_and_config 2024-03-25T18:32:12.796 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.794+0000 7fb64b47b640 10 monclient: build_initial_monmap 2024-03-25T18:32:12.796 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.794+0000 7fb64b47b640 10 monclient: monmap: 2024-03-25T18:32:12.796 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:12.796 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:12.796 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:32:12.795921+0000 2024-03-25T18:32:12.796 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:32:12.795921+0000 2024-03-25T18:32:12.796 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:12.797 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:12.797 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:12.797 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:32:12.797 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:32:12.797 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:12.798 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.795+0000 7fb64b47b640 1 Processor -- start 2024-03-25T18:32:12.798 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.795+0000 7fb64b47b640 1 -- start start 2024-03-25T18:32:12.798 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.795+0000 7fb64b47b640 10 monclient: init 2024-03-25T18:32:12.798 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.796+0000 7fb64b47b640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:12.798 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.796+0000 7fb64b47b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:12.799 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.796+0000 7fb64b47b640 10 monclient(hunting): picked mon.noname-a con 0x7fb6440a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:12.799 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.796+0000 7fb64b47b640 10 monclient(hunting): picked mon.noname-b con 0x7fb64413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:12.799 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.796+0000 7fb64b47b640 10 monclient(hunting): picked mon.noname-c con 0x7fb6441430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:12.799 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.796+0000 7fb64b47b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb644142520 con 0x7fb6440a98f0 2024-03-25T18:32:12.799 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.796+0000 7fb64b47b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6440b28a0 con 0x7fb6441430c0 2024-03-25T18:32:12.799 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.796+0000 7fb64b47b640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6440b2a30 con 0x7fb64413f710 2024-03-25T18:32:12.799 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.796+0000 7fb64b47b640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:12.799 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.796+0000 7fb64b47b640 10 monclient(hunting): authenticate will time out at 7750.964562s 2024-03-25T18:32:12.804 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.803+0000 7fb648a16640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7fb64413f710 0x7fb6441429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:33202/0 (socket says 172.21.15.99:33202) 2024-03-25T18:32:12.804 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.803+0000 7fb648a16640 1 -- 172.21.15.99:0/352712282 learned_addr learned my addr 172.21.15.99:0/352712282 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:12.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.826+0000 7fb63bfff640 1 -- 172.21.15.99:0/352712282 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500478472 0 0) 0x7fb6440b2a30 con 0x7fb64413f710 2024-03-25T18:32:12.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.826+0000 7fb63bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:12.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.826+0000 7fb63bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:12.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.826+0000 7fb63bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:12.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.826+0000 7fb63bfff640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:32:12.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.826+0000 7fb63bfff640 1 -- 172.21.15.99:0/352712282 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb630003520 con 0x7fb64413f710 2024-03-25T18:32:12.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.826+0000 7fb63bfff640 1 -- 172.21.15.99:0/352712282 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760240742 0 0) 0x7fb644142520 con 0x7fb6440a98f0 2024-03-25T18:32:12.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.826+0000 7fb63bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:12.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.826+0000 7fb63bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:12.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.826+0000 7fb63bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:12.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.826+0000 7fb63bfff640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:32:12.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.826+0000 7fb63bfff640 1 -- 172.21.15.99:0/352712282 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6440b2a30 con 0x7fb6440a98f0 2024-03-25T18:32:12.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 1 -- 172.21.15.99:0/352712282 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1458928759 0 0) 0x7fb6440b2a30 con 0x7fb6440a98f0 2024-03-25T18:32:12.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 1 -- 172.21.15.99:0/352712282 >> v1:172.21.15.99:6790/0 conn(0x7fb64413f710 legacy=0x7fb6441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:12.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 1 -- 172.21.15.99:0/352712282 >> v1:172.21.15.190:6789/0 conn(0x7fb6441430c0 legacy=0x7fb64414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:12.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:12.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 1 monclient: found mon.noname-a 2024-03-25T18:32:12.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:12.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:32:12.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 1 -- 172.21.15.99:0/352712282 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6440b44c0 con 0x7fb6440a98f0 2024-03-25T18:32:12.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 10 monclient: _finish_auth 0 2024-03-25T18:32:12.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:12.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:12.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 1 -- 172.21.15.99:0/352712282 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb640001670 con 0x7fb6440a98f0 2024-03-25T18:32:12.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:12.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:12.850 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb63bfff640 10 monclient: dump: 2024-03-25T18:32:12.850 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:12.850 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:12.850 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:12.850 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:12.850 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:12.850 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:12.850 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:12.850 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:12.850 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:12.850 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:12.850 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb64b47b640 5 monclient: authenticate success, global_id 4518 2024-03-25T18:32:12.850 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.847+0000 7fb64b47b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:12.909 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.908+0000 7fb63bfff640 1 -- 172.21.15.99:0/352712282 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6400017d0 con 0x7fb6440a98f0 2024-03-25T18:32:12.909 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.908+0000 7fb63bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:12.910 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.908+0000 7fb63bfff640 1 -- 172.21.15.99:0/352712282 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb640001be0 con 0x7fb6440a98f0 2024-03-25T18:32:12.910 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.908+0000 7fb64b47b640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:12.910 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.908+0000 7fb64b47b640 4 set_mon_vals no callback set 2024-03-25T18:32:12.910 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.908+0000 7fb63bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:12.910 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.908+0000 7fb63bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:12.910 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.908+0000 7fb63bfff640 10 monclient: dump: 2024-03-25T18:32:12.910 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:12.910 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:12.910 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:12.910 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:12.910 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:12.910 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:12.910 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:12.911 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:12.911 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:12.911 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:12.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.909+0000 7fb64b47b640 10 monclient: shutdown 2024-03-25T18:32:12.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.909+0000 7fb64b47b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:12.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.909+0000 7fb64b47b640 1 -- 172.21.15.99:0/352712282 >> v1:172.21.15.99:6789/0 conn(0x7fb6440a98f0 legacy=0x7fb6440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:12.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.909+0000 7fb64b47b640 1 -- 172.21.15.99:0/352712282 shutdown_connections 2024-03-25T18:32:12.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.909+0000 7fb64b47b640 1 -- 172.21.15.99:0/352712282 >> 172.21.15.99:0/352712282 conn(0x7fb6440b7ea0 msgr2=0x7fb6440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:12.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.909+0000 7fb64b47b640 1 -- 172.21.15.99:0/352712282 shutdown_connections 2024-03-25T18:32:12.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.909+0000 7fb64b47b640 1 -- 172.21.15.99:0/352712282 wait complete. 2024-03-25T18:32:12.914 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.910+0000 7fb64b47b640 10 monclient: build_initial_monmap 2024-03-25T18:32:12.914 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.910+0000 7fb64b47b640 10 monclient: monmap: 2024-03-25T18:32:12.914 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:12.914 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:12.914 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:32:12.914 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:32:12.914 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:12.914 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:32:12.914 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:12.914 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:32:12.914 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:32:12.914 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.910+0000 7fb64b47b640 1 librados: starting msgr at 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.910+0000 7fb64b47b640 1 librados: starting objecter 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.910+0000 7fb64b47b640 1 Processor -- start 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.910+0000 7fb64b47b640 1 -- start start 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.910+0000 7fb64b47b640 1 librados: setting wanted keys 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.910+0000 7fb64b47b640 1 librados: calling monclient init 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.910+0000 7fb64b47b640 10 monclient: init 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.911+0000 7fb64b47b640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.911+0000 7fb64b47b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.911+0000 7fb64b47b640 10 monclient(hunting): picked mon.noname-a con 0x7fb6440a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.911+0000 7fb64b47b640 10 monclient(hunting): picked mon.noname-c con 0x7fb64413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.911+0000 7fb64b47b640 10 monclient(hunting): picked mon.noname-b con 0x7fb6441430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.911+0000 7fb64b47b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6440b0fe0 con 0x7fb6440a98f0 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.911+0000 7fb64b47b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6441d48f0 con 0x7fb6441430c0 2024-03-25T18:32:12.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.911+0000 7fb64b47b640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6441d4e40 con 0x7fb64413f710 2024-03-25T18:32:12.916 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.911+0000 7fb64b47b640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:12.916 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.911+0000 7fb64b47b640 10 monclient(hunting): authenticate will time out at 7751.079130s 2024-03-25T18:32:12.916 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.913+0000 7fb649a18640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb6441430c0 0x7fb64414d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:40846/0 (socket says 172.21.15.99:40846) 2024-03-25T18:32:12.916 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.913+0000 7fb649a18640 1 -- 172.21.15.99:0/1620941301 learned_addr learned my addr 172.21.15.99:0/1620941301 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:12.919 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.918+0000 7fb639ffb640 10 client.?.objecter ms_handle_connect 0x7fb6441430c0 2024-03-25T18:32:12.919 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.918+0000 7fb639ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:12.928 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.927+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398035017 0 0) 0x7fb6441d48f0 con 0x7fb6441430c0 2024-03-25T18:32:12.928 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.927+0000 7fb639ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:12.928 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.927+0000 7fb639ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:12.928 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.927+0000 7fb639ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:12.928 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.927+0000 7fb639ffb640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:32:12.928 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.927+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb628004eb0 con 0x7fb6441430c0 2024-03-25T18:32:12.938 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.937+0000 7fb639ffb640 10 client.?.objecter ms_handle_connect 0x7fb6440a98f0 2024-03-25T18:32:12.938 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.937+0000 7fb639ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:12.946 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.945+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3629905031 0 0) 0x7fb6440b0fe0 con 0x7fb6440a98f0 2024-03-25T18:32:12.946 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.945+0000 7fb639ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:12.946 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.945+0000 7fb639ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:12.947 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.945+0000 7fb639ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:12.947 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.945+0000 7fb639ffb640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:32:12.947 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.945+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6441d48f0 con 0x7fb6440a98f0 2024-03-25T18:32:12.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.950+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3485056705 0 0) 0x7fb628004eb0 con 0x7fb6441430c0 2024-03-25T18:32:12.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.950+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6440b0fe0 con 0x7fb6441430c0 2024-03-25T18:32:12.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.950+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb63c001c30 con 0x7fb6441430c0 2024-03-25T18:32:12.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.950+0000 7fb639ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:12.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.950+0000 7fb639ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:12.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.950+0000 7fb639ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:12.952 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:12.952 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:12.952 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:12.952 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:12.952 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:12.952 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:12.952 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:12.952 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:12.952 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:12.952 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:12.952 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.950+0000 7fb639ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:12.957 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.956+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 9009066 0 0) 0x7fb6441d48f0 con 0x7fb6440a98f0 2024-03-25T18:32:12.957 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.956+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb628004eb0 con 0x7fb6440a98f0 2024-03-25T18:32:12.957 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.956+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb640001ae0 con 0x7fb6440a98f0 2024-03-25T18:32:12.957 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.956+0000 7fb639ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:12.957 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.956+0000 7fb639ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:12.958 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.956+0000 7fb639ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:12.958 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:12.958 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:12.958 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:12.958 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:12.958 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:12.958 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:12.958 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:12.958 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:12.958 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:12.958 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:12.961 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.960+0000 7fb639ffb640 10 client.?.objecter ms_handle_connect 0x7fb64413f710 2024-03-25T18:32:12.961 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.960+0000 7fb639ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:12.971 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1393640318 0 0) 0x7fb6440b0fe0 con 0x7fb6441430c0 2024-03-25T18:32:12.971 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 >> v1:172.21.15.99:6790/0 conn(0x7fb64413f710 legacy=0x7fb6441d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:12.971 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 >> v1:172.21.15.99:6789/0 conn(0x7fb6440a98f0 legacy=0x7fb6441d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:12.971 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb639ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:32:12.971 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb639ffb640 1 monclient: found mon.b 2024-03-25T18:32:12.971 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb639ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:12.971 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb639ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:12.971 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6441d4fd0 con 0x7fb6441430c0 2024-03-25T18:32:12.971 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb639ffb640 10 monclient: _finish_auth 0 2024-03-25T18:32:12.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb639ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:32:12.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb639ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:12.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb64b47b640 5 monclient: authenticate success, global_id 4534 2024-03-25T18:32:12.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb64b47b640 10 monclient: _renew_subs 2024-03-25T18:32:12.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb64b47b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:12.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.969+0000 7fb64b47b640 1 -- 172.21.15.99:0/1620941301 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64414e340 con 0x7fb6441430c0 2024-03-25T18:32:12.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.970+0000 7fb64b47b640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:12.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.970+0000 7fb64b47b640 10 monclient: _renew_subs 2024-03-25T18:32:12.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.970+0000 7fb64b47b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:12.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.970+0000 7fb64b47b640 1 -- 172.21.15.99:0/1620941301 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6441d7ce0 con 0x7fb6441430c0 2024-03-25T18:32:12.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.970+0000 7fb64b47b640 1 librados: init done 2024-03-25T18:32:12.973 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.971+0000 7fb64b47b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:12.973 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.971+0000 7fb64b47b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:12.973 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.971+0000 7fb64b47b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:12.973 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:12.971+0000 7fb64b47b640 1 -- 172.21.15.99:0/1620941301 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6441d4ae0 con 0x7fb6441430c0 2024-03-25T18:32:13.280 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.279+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb63c003990 con 0x7fb6441430c0 2024-03-25T18:32:13.280 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.279+0000 7fb639ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:13.280 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.279+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb63c003da0 con 0x7fb6441430c0 2024-03-25T18:32:13.280 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.279+0000 7fb639ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:13.280 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.279+0000 7fb639ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:13.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.279+0000 7fb639ffb640 10 monclient: dump: 2024-03-25T18:32:13.281 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:13.281 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:13.281 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:13.281 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:13.281 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:13.281 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:13.281 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:13.281 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:13.281 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:13.281 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:13.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.279+0000 7fb63b7fe640 4 set_mon_vals no callback set 2024-03-25T18:32:13.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.280+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fb63c003fc0 con 0x7fb6441430c0 2024-03-25T18:32:13.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.280+0000 7fb649217640 10 monclient: get_auth_request con 0x7fb62806de40 auth_method 0 2024-03-25T18:32:13.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.281+0000 7fb639ffb640 10 client.4534.objecter ms_handle_connect 0x7fb62806de40 2024-03-25T18:32:13.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.287+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fb63c054a50 con 0x7fb6441430c0 2024-03-25T18:32:13.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.287+0000 7fb639ffb640 10 client.4534.objecter ms_dispatch 0x7fb644000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:13.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.287+0000 7fb639ffb640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:13.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.287+0000 7fb639ffb640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:13.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.287+0000 7fb639ffb640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-25T18:32:13.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.288+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb63c050560 con 0x7fb6441430c0 2024-03-25T18:32:13.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.288+0000 7fb639ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:13.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.288+0000 7fb639ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:13.542 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.541+0000 7fb64b47b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:13.542 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.541+0000 7fb64b47b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:13.542 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.541+0000 7fb64b47b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:13.542 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.541+0000 7fb64b47b640 1 -- 172.21.15.99:0/1620941301 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6440627b0 con 0x7fb6441430c0 2024-03-25T18:32:13.575 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.573+0000 7fb639ffb640 1 -- 172.21.15.99:0/1620941301 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fb63c052490 con 0x7fb6441430c0 2024-03-25T18:32:13.575 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.573+0000 7fb639ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:13.575 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.573+0000 7fb639ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:13.575 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:32:13.575 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:13.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.576+0000 7fb64b47b640 10 librados: watch_flush enter 2024-03-25T18:32:13.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.576+0000 7fb64b47b640 10 librados: watch_flush exit 2024-03-25T18:32:13.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.576+0000 7fb64b47b640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-25T18:32:13.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.576+0000 7fb64b47b640 10 client.4534.objecter successfully canceled tick 2024-03-25T18:32:13.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.576+0000 7fb64b47b640 1 -- 172.21.15.99:0/1620941301 >> v1:172.21.15.190:6800/4220409699 conn(0x7fb62806de40 legacy=0x7fb628070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:13.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.576+0000 7fb64b47b640 10 monclient: shutdown 2024-03-25T18:32:13.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.576+0000 7fb64b47b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:13.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.576+0000 7fb64b47b640 1 -- 172.21.15.99:0/1620941301 >> v1:172.21.15.190:6789/0 conn(0x7fb6441430c0 legacy=0x7fb64414d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:13.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.576+0000 7fb64b47b640 1 -- 172.21.15.99:0/1620941301 shutdown_connections 2024-03-25T18:32:13.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.576+0000 7fb64b47b640 1 -- 172.21.15.99:0/1620941301 >> 172.21.15.99:0/1620941301 conn(0x7fb6440b7ea0 msgr2=0x7fb644074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:13.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.576+0000 7fb64b47b640 1 -- 172.21.15.99:0/1620941301 shutdown_connections 2024-03-25T18:32:13.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.576+0000 7fb64b47b640 1 -- 172.21.15.99:0/1620941301 wait complete. 2024-03-25T18:32:13.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:13.576+0000 7fb64b47b640 1 librados: shutdown 2024-03-25T18:32:13.590 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:20.397 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:20.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.509+0000 7f178eb0d640 10 monclient: get_monmap_and_config 2024-03-25T18:32:20.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.509+0000 7f178eb0d640 10 monclient: build_initial_monmap 2024-03-25T18:32:20.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.509+0000 7f178eb0d640 10 monclient: monmap: 2024-03-25T18:32:20.510 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:20.510 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:20.510 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:32:20.510290+0000 2024-03-25T18:32:20.510 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:32:20.510290+0000 2024-03-25T18:32:20.510 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:20.511 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:20.511 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:20.511 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:32:20.511 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:32:20.511 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:20.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.510+0000 7f178eb0d640 1 Processor -- start 2024-03-25T18:32:20.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.510+0000 7f178eb0d640 1 -- start start 2024-03-25T18:32:20.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.510+0000 7f178eb0d640 10 monclient: init 2024-03-25T18:32:20.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.510+0000 7f178eb0d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:20.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.510+0000 7f178eb0d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:20.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.510+0000 7f178eb0d640 10 monclient(hunting): picked mon.noname-a con 0x7f17880b4d60 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:20.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.510+0000 7f178eb0d640 10 monclient(hunting): picked mon.noname-c con 0x7f17880b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:20.513 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.510+0000 7f178eb0d640 10 monclient(hunting): picked mon.noname-b con 0x7f178814be90 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:20.513 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.510+0000 7f178eb0d640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17880af940 con 0x7f17880b4d60 2024-03-25T18:32:20.513 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.510+0000 7f178eb0d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17880b0a70 con 0x7f17880b1730 2024-03-25T18:32:20.513 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.510+0000 7f178eb0d640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17880b0d50 con 0x7f178814be90 2024-03-25T18:32:20.513 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.510+0000 7f178eb0d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:20.513 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.510+0000 7f178eb0d640 10 monclient(hunting): authenticate will time out at 7758.678730s 2024-03-25T18:32:20.519 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.518+0000 7f1787fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f17880b1730 0x7f17880b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:58440/0 (socket says 172.21.15.99:58440) 2024-03-25T18:32:20.519 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.518+0000 7f1787fff640 1 -- 172.21.15.99:0/2923024143 learned_addr learned my addr 172.21.15.99:0/2923024143 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:20.528 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.527+0000 7f17877fe640 1 -- 172.21.15.99:0/2923024143 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181601900 0 0) 0x7f17880b0d50 con 0x7f178814be90 2024-03-25T18:32:20.528 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.527+0000 7f17877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:20.528 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.527+0000 7f17877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:20.528 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.527+0000 7f17877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:20.528 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.527+0000 7f17877fe640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:32:20.528 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.527+0000 7f17877fe640 1 -- 172.21.15.99:0/2923024143 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1778003520 con 0x7f178814be90 2024-03-25T18:32:20.532 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.531+0000 7f17877fe640 1 -- 172.21.15.99:0/2923024143 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680205872 0 0) 0x7f17880b0a70 con 0x7f17880b1730 2024-03-25T18:32:20.533 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.531+0000 7f17877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:20.533 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.531+0000 7f17877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:20.533 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.531+0000 7f17877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:20.533 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.531+0000 7f17877fe640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:32:20.533 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.531+0000 7f17877fe640 1 -- 172.21.15.99:0/2923024143 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17880b0d50 con 0x7f17880b1730 2024-03-25T18:32:20.539 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 1 -- 172.21.15.99:0/2923024143 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1623486709 0 0) 0x7f17880b0d50 con 0x7f17880b1730 2024-03-25T18:32:20.539 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 1 -- 172.21.15.99:0/2923024143 >> v1:172.21.15.99:6790/0 conn(0x7f178814be90 legacy=0x7f178814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 1 -- 172.21.15.99:0/2923024143 >> v1:172.21.15.99:6789/0 conn(0x7f17880b4d60 legacy=0x7f178813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 1 monclient: found mon.noname-c 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 1 -- 172.21.15.99:0/2923024143 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17880b1030 con 0x7f17880b1730 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 1 -- 172.21.15.99:0/2923024143 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f1774001670 con 0x7f17880b1730 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f17877fe640 10 monclient: dump: 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:20.540 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:20.541 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:20.541 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:20.541 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:20.541 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:20.541 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:20.541 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:20.541 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:20.541 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f178eb0d640 5 monclient: authenticate success, global_id 4537 2024-03-25T18:32:20.541 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.538+0000 7f178eb0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:20.545 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.544+0000 7f17877fe640 1 -- 172.21.15.99:0/2923024143 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17740017d0 con 0x7f17880b1730 2024-03-25T18:32:20.545 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.544+0000 7f17877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:20.545 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.544+0000 7f17877fe640 1 -- 172.21.15.99:0/2923024143 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f1774001ba0 con 0x7f17880b1730 2024-03-25T18:32:20.545 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.544+0000 7f17877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:20.545 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.544+0000 7f17877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:20.545 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.544+0000 7f17877fe640 10 monclient: dump: 2024-03-25T18:32:20.545 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:20.545 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:20.545 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:20.545 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:20.545 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:20.546 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:20.546 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:20.546 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:20.546 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:20.546 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:20.546 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.544+0000 7f178eb0d640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:20.546 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.544+0000 7f178eb0d640 4 set_mon_vals no callback set 2024-03-25T18:32:20.547 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.545+0000 7f178eb0d640 10 monclient: shutdown 2024-03-25T18:32:20.547 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.545+0000 7f178eb0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.545+0000 7f178eb0d640 1 -- 172.21.15.99:0/2923024143 >> v1:172.21.15.190:6789/0 conn(0x7f17880b1730 legacy=0x7f17880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.545+0000 7f178eb0d640 1 -- 172.21.15.99:0/2923024143 shutdown_connections 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.545+0000 7f178eb0d640 1 -- 172.21.15.99:0/2923024143 >> 172.21.15.99:0/2923024143 conn(0x7f17880617a0 msgr2=0x7f1788061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.545+0000 7f178eb0d640 1 -- 172.21.15.99:0/2923024143 shutdown_connections 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.545+0000 7f178eb0d640 1 -- 172.21.15.99:0/2923024143 wait complete. 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.546+0000 7f178eb0d640 10 monclient: build_initial_monmap 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.546+0000 7f178eb0d640 10 monclient: monmap: 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:20.548 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:32:20.549 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:32:20.549 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:20.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.546+0000 7f178eb0d640 1 librados: starting msgr at 2024-03-25T18:32:20.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.546+0000 7f178eb0d640 1 librados: starting objecter 2024-03-25T18:32:20.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.546+0000 7f178eb0d640 1 Processor -- start 2024-03-25T18:32:20.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.546+0000 7f178eb0d640 1 -- start start 2024-03-25T18:32:20.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.546+0000 7f178eb0d640 1 librados: setting wanted keys 2024-03-25T18:32:20.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.546+0000 7f178eb0d640 1 librados: calling monclient init 2024-03-25T18:32:20.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.546+0000 7f178eb0d640 10 monclient: init 2024-03-25T18:32:20.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.547+0000 7f178eb0d640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:20.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.547+0000 7f178eb0d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:20.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.547+0000 7f178eb0d640 10 monclient(hunting): picked mon.noname-c con 0x7f17880b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:20.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.547+0000 7f178eb0d640 10 monclient(hunting): picked mon.noname-b con 0x7f17880b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:20.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.547+0000 7f178eb0d640 10 monclient(hunting): picked mon.noname-a con 0x7f178814be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:20.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.547+0000 7f178eb0d640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17880a9d70 con 0x7f178814be90 2024-03-25T18:32:20.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.547+0000 7f178eb0d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17880aac20 con 0x7f17880b4d60 2024-03-25T18:32:20.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.547+0000 7f178eb0d640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17880aaf00 con 0x7f17880b1730 2024-03-25T18:32:20.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.547+0000 7f178eb0d640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:20.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.547+0000 7f178eb0d640 10 monclient(hunting): authenticate will time out at 7758.715439s 2024-03-25T18:32:20.557 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.554+0000 7f1787fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f17880b4d60 0x7f1788140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:58452/0 (socket says 172.21.15.99:58452) 2024-03-25T18:32:20.557 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.554+0000 7f1787fff640 1 -- 172.21.15.99:0/1716532341 learned_addr learned my addr 172.21.15.99:0/1716532341 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:20.558 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.557+0000 7f17857fa640 10 client.?.objecter ms_handle_connect 0x7f17880b4d60 2024-03-25T18:32:20.558 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.557+0000 7f17857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:20.570 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.569+0000 7f17857fa640 10 client.?.objecter ms_handle_connect 0x7f178814be90 2024-03-25T18:32:20.570 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.569+0000 7f17857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:20.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.571+0000 7f17857fa640 10 client.?.objecter ms_handle_connect 0x7f17880b1730 2024-03-25T18:32:20.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.571+0000 7f17857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:20.576 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.575+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950912949 0 0) 0x7f17880aac20 con 0x7f17880b4d60 2024-03-25T18:32:20.576 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.575+0000 7f17857fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:20.576 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.575+0000 7f17857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:20.576 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.575+0000 7f17857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:20.576 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.575+0000 7f17857fa640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:32:20.576 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.575+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f176c004eb0 con 0x7f17880b4d60 2024-03-25T18:32:20.582 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.581+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309404824 0 0) 0x7f17880a9d70 con 0x7f178814be90 2024-03-25T18:32:20.582 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.581+0000 7f17857fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:20.582 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.581+0000 7f17857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:20.582 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.581+0000 7f17857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:20.582 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.581+0000 7f17857fa640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:32:20.582 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.581+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17880aac20 con 0x7f178814be90 2024-03-25T18:32:20.586 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.585+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1940840955 0 0) 0x7f176c004eb0 con 0x7f17880b4d60 2024-03-25T18:32:20.586 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.585+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17880a9d70 con 0x7f17880b4d60 2024-03-25T18:32:20.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.587+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709157328 0 0) 0x7f17880aaf00 con 0x7f17880b1730 2024-03-25T18:32:20.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.587+0000 7f17857fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:20.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.587+0000 7f17857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:20.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.587+0000 7f17857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:20.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.587+0000 7f17857fa640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:32:20.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.587+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f176c004eb0 con 0x7f17880b1730 2024-03-25T18:32:20.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f17740047e0 con 0x7f17880b4d60 2024-03-25T18:32:20.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:20.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:20.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 10 monclient(hunting): dump: 2024-03-25T18:32:20.601 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:20.601 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:20.602 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:20.602 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:20.602 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:20.602 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:20.602 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:20.602 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:20.602 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:20.602 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:20.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:20.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1675066567 0 0) 0x7f17880a9d70 con 0x7f17880b4d60 2024-03-25T18:32:20.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 >> v1:172.21.15.99:6790/0 conn(0x7f17880b1730 legacy=0x7f1788142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:20.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 >> v1:172.21.15.99:6789/0 conn(0x7f178814be90 legacy=0x7f17881e27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:20.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 10 monclient: _finish_hunting 0 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 1 monclient: found mon.b 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17880a9850 con 0x7f17880b4d60 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 10 monclient: _finish_auth 0 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 10 monclient: _check_auth_tickets 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f17857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f178eb0d640 5 monclient: authenticate success, global_id 4540 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f178eb0d640 10 monclient: _renew_subs 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f178eb0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f178eb0d640 1 -- 172.21.15.99:0/1716532341 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17880a9470 con 0x7f17880b4d60 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f178eb0d640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f178eb0d640 10 monclient: _renew_subs 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f178eb0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f178eb0d640 1 -- 172.21.15.99:0/1716532341 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1788140c70 con 0x7f17880b4d60 2024-03-25T18:32:20.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.599+0000 7f178eb0d640 1 librados: init done 2024-03-25T18:32:20.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.600+0000 7f178eb0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:20.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.600+0000 7f178eb0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:20.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.600+0000 7f178eb0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:20.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.600+0000 7f178eb0d640 1 -- 172.21.15.99:0/1716532341 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17880a96b0 con 0x7f17880b4d60 2024-03-25T18:32:20.875 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.873+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1774004bf0 con 0x7f17880b4d60 2024-03-25T18:32:20.875 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.873+0000 7f17857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:20.875 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.873+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f1774004d60 con 0x7f17880b4d60 2024-03-25T18:32:20.875 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.873+0000 7f17857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:20.875 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.873+0000 7f17857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:20.875 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.873+0000 7f17857fa640 10 monclient: dump: 2024-03-25T18:32:20.875 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:20.875 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:20.875 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:20.875 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:20.875 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:20.876 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:20.876 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:20.876 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:20.876 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:20.876 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:20.876 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.873+0000 7f1786ffd640 4 set_mon_vals no callback set 2024-03-25T18:32:20.876 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.874+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f177401ad30 con 0x7f17880b4d60 2024-03-25T18:32:20.877 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.875+0000 7f178c8a9640 10 monclient: get_auth_request con 0x7f176c06de60 auth_method 0 2024-03-25T18:32:20.877 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.875+0000 7f17857fa640 10 client.4540.objecter ms_handle_connect 0x7f176c06de60 2024-03-25T18:32:20.880 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.879+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f17740547f0 con 0x7f17880b4d60 2024-03-25T18:32:20.880 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.879+0000 7f17857fa640 10 client.4540.objecter ms_dispatch 0x7f1788000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:20.880 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.879+0000 7f17857fa640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:20.880 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.879+0000 7f17857fa640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:20.880 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.879+0000 7f17857fa640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-25T18:32:20.881 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.879+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1774050300 con 0x7f17880b4d60 2024-03-25T18:32:20.881 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.879+0000 7f17857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:20.881 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:20.879+0000 7f17857fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:21.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.110+0000 7f178eb0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:21.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.110+0000 7f178eb0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:21.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.110+0000 7f178eb0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:21.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.110+0000 7f178eb0d640 1 -- 172.21.15.99:0/1716532341 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17880a99e0 con 0x7f17880b4d60 2024-03-25T18:32:21.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.138+0000 7f17857fa640 1 -- 172.21.15.99:0/1716532341 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f1774052230 con 0x7f17880b4d60 2024-03-25T18:32:21.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.138+0000 7f17857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:21.139 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.138+0000 7f17857fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:21.140 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:32:21.140 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:21.142 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.141+0000 7f178eb0d640 10 librados: watch_flush enter 2024-03-25T18:32:21.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.141+0000 7f178eb0d640 10 librados: watch_flush exit 2024-03-25T18:32:21.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.141+0000 7f178eb0d640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-25T18:32:21.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.141+0000 7f178eb0d640 10 client.4540.objecter successfully canceled tick 2024-03-25T18:32:21.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.141+0000 7f178eb0d640 1 -- 172.21.15.99:0/1716532341 >> v1:172.21.15.190:6800/4220409699 conn(0x7f176c06de60 legacy=0x7f176c070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.141+0000 7f178eb0d640 10 monclient: shutdown 2024-03-25T18:32:21.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.141+0000 7f178eb0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:21.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.141+0000 7f178eb0d640 1 -- 172.21.15.99:0/1716532341 >> v1:172.21.15.190:6789/0 conn(0x7f17880b4d60 legacy=0x7f1788140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.141+0000 7f178eb0d640 1 -- 172.21.15.99:0/1716532341 shutdown_connections 2024-03-25T18:32:21.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.141+0000 7f178eb0d640 1 -- 172.21.15.99:0/1716532341 >> 172.21.15.99:0/1716532341 conn(0x7f17880617a0 msgr2=0x7f178813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:21.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.141+0000 7f178eb0d640 1 -- 172.21.15.99:0/1716532341 shutdown_connections 2024-03-25T18:32:21.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.141+0000 7f178eb0d640 1 -- 172.21.15.99:0/1716532341 wait complete. 2024-03-25T18:32:21.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:21.141+0000 7f178eb0d640 1 librados: shutdown 2024-03-25T18:32:21.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:27.961 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:28.075 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.074+0000 7fdad39cf640 10 monclient: get_monmap_and_config 2024-03-25T18:32:28.075 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.074+0000 7fdad39cf640 10 monclient: build_initial_monmap 2024-03-25T18:32:28.075 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.074+0000 7fdad39cf640 10 monclient: monmap: 2024-03-25T18:32:28.076 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:28.076 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:28.076 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:32:28.075350+0000 2024-03-25T18:32:28.076 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:32:28.075350+0000 2024-03-25T18:32:28.076 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:28.076 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:28.076 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:28.076 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:32:28.076 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:32:28.076 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:28.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.075+0000 7fdad39cf640 1 Processor -- start 2024-03-25T18:32:28.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.075+0000 7fdad39cf640 1 -- start start 2024-03-25T18:32:28.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.075+0000 7fdad39cf640 10 monclient: init 2024-03-25T18:32:28.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.075+0000 7fdad39cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:28.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.075+0000 7fdad39cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:28.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.075+0000 7fdad39cf640 10 monclient(hunting): picked mon.noname-a con 0x7fdacc0a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:28.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.075+0000 7fdad39cf640 10 monclient(hunting): picked mon.noname-b con 0x7fdacc13f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:28.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.075+0000 7fdad39cf640 10 monclient(hunting): picked mon.noname-c con 0x7fdacc1430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:28.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.075+0000 7fdad39cf640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc142520 con 0x7fdacc0a98f0 2024-03-25T18:32:28.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.075+0000 7fdad39cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc0b28a0 con 0x7fdacc1430c0 2024-03-25T18:32:28.078 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.075+0000 7fdad39cf640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc0b2a30 con 0x7fdacc13f710 2024-03-25T18:32:28.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.075+0000 7fdad39cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:28.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.076+0000 7fdad39cf640 10 monclient(hunting): authenticate will time out at 7766.243878s 2024-03-25T18:32:28.090 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.088+0000 7fdad176b640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fdacc0a98f0 0x7fdacc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:42382/0 (socket says 172.21.15.99:42382) 2024-03-25T18:32:28.090 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.088+0000 7fdad176b640 1 -- 172.21.15.99:0/3091037056 learned_addr learned my addr 172.21.15.99:0/3091037056 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:28.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.098+0000 7fdac3fff640 1 -- 172.21.15.99:0/3091037056 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455435515 0 0) 0x7fdacc142520 con 0x7fdacc0a98f0 2024-03-25T18:32:28.100 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.098+0000 7fdac3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:28.100 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.098+0000 7fdac3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:28.100 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.098+0000 7fdac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:28.100 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.098+0000 7fdac3fff640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:32:28.100 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.098+0000 7fdac3fff640 1 -- 172.21.15.99:0/3091037056 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab8003520 con 0x7fdacc0a98f0 2024-03-25T18:32:28.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.107+0000 7fdac3fff640 1 -- 172.21.15.99:0/3091037056 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922383360 0 0) 0x7fdacc0b2a30 con 0x7fdacc13f710 2024-03-25T18:32:28.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.107+0000 7fdac3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:28.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.107+0000 7fdac3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:28.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.107+0000 7fdac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:28.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.107+0000 7fdac3fff640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:32:28.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.107+0000 7fdac3fff640 1 -- 172.21.15.99:0/3091037056 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdacc142520 con 0x7fdacc13f710 2024-03-25T18:32:28.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.113+0000 7fdac3fff640 1 -- 172.21.15.99:0/3091037056 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2575208049 0 0) 0x7fdab8003520 con 0x7fdacc0a98f0 2024-03-25T18:32:28.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.113+0000 7fdac3fff640 1 -- 172.21.15.99:0/3091037056 >> v1:172.21.15.99:6790/0 conn(0x7fdacc13f710 legacy=0x7fdacc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:28.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.113+0000 7fdac3fff640 1 -- 172.21.15.99:0/3091037056 >> v1:172.21.15.190:6789/0 conn(0x7fdacc1430c0 legacy=0x7fdacc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:28.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.114+0000 7fdac3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:28.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.114+0000 7fdac3fff640 1 monclient: found mon.noname-a 2024-03-25T18:32:28.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.114+0000 7fdac3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:28.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.114+0000 7fdac3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:32:28.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.114+0000 7fdac3fff640 1 -- 172.21.15.99:0/3091037056 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc0b44c0 con 0x7fdacc0a98f0 2024-03-25T18:32:28.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.114+0000 7fdac3fff640 10 monclient: _finish_auth 0 2024-03-25T18:32:28.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.114+0000 7fdac3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:28.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.114+0000 7fdac3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:28.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.114+0000 7fdac3fff640 1 -- 172.21.15.99:0/3091037056 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fdac8001690 con 0x7fdacc0a98f0 2024-03-25T18:32:28.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.114+0000 7fdac3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:28.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.114+0000 7fdac3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:28.116 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.114+0000 7fdac3fff640 10 monclient: dump: 2024-03-25T18:32:28.116 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:28.116 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:28.116 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:28.116 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:28.116 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:28.116 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:28.116 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:28.117 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:28.117 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:28.117 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:28.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.114+0000 7fdad39cf640 5 monclient: authenticate success, global_id 4530 2024-03-25T18:32:28.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.114+0000 7fdad39cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:28.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.116+0000 7fdac3fff640 1 -- 172.21.15.99:0/3091037056 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdac80017f0 con 0x7fdacc0a98f0 2024-03-25T18:32:28.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.116+0000 7fdac3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:28.117 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.116+0000 7fdac3fff640 1 -- 172.21.15.99:0/3091037056 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fdac8001c00 con 0x7fdacc0a98f0 2024-03-25T18:32:28.118 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.116+0000 7fdac3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:28.118 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.116+0000 7fdac3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:28.118 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.116+0000 7fdac3fff640 10 monclient: dump: 2024-03-25T18:32:28.118 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:28.118 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:28.118 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:28.118 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:28.118 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:28.118 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:28.118 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:28.118 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:28.118 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:28.118 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:28.118 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.116+0000 7fdad39cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:28.119 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.116+0000 7fdad39cf640 4 set_mon_vals no callback set 2024-03-25T18:32:28.120 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.117+0000 7fdad39cf640 10 monclient: shutdown 2024-03-25T18:32:28.120 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.117+0000 7fdad39cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:28.121 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.117+0000 7fdad39cf640 1 -- 172.21.15.99:0/3091037056 >> v1:172.21.15.99:6789/0 conn(0x7fdacc0a98f0 legacy=0x7fdacc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:28.121 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.117+0000 7fdad39cf640 1 -- 172.21.15.99:0/3091037056 shutdown_connections 2024-03-25T18:32:28.121 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.117+0000 7fdad39cf640 1 -- 172.21.15.99:0/3091037056 >> 172.21.15.99:0/3091037056 conn(0x7fdacc0b7ea0 msgr2=0x7fdacc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:28.121 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.117+0000 7fdad39cf640 1 -- 172.21.15.99:0/3091037056 shutdown_connections 2024-03-25T18:32:28.121 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.117+0000 7fdad39cf640 1 -- 172.21.15.99:0/3091037056 wait complete. 2024-03-25T18:32:28.121 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.118+0000 7fdad39cf640 10 monclient: build_initial_monmap 2024-03-25T18:32:28.121 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.118+0000 7fdad39cf640 10 monclient: monmap: 2024-03-25T18:32:28.121 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:28.121 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:28.121 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:32:28.121 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:32:28.121 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:28.121 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:32:28.122 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:28.122 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:32:28.122 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:32:28.122 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:28.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.118+0000 7fdad39cf640 1 librados: starting msgr at 2024-03-25T18:32:28.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.118+0000 7fdad39cf640 1 librados: starting objecter 2024-03-25T18:32:28.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.118+0000 7fdad39cf640 1 Processor -- start 2024-03-25T18:32:28.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.118+0000 7fdad39cf640 1 -- start start 2024-03-25T18:32:28.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.118+0000 7fdad39cf640 1 librados: setting wanted keys 2024-03-25T18:32:28.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.118+0000 7fdad39cf640 1 librados: calling monclient init 2024-03-25T18:32:28.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.118+0000 7fdad39cf640 10 monclient: init 2024-03-25T18:32:28.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.119+0000 7fdad39cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:28.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.119+0000 7fdad39cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:32:28.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.119+0000 7fdad39cf640 10 monclient(hunting): picked mon.noname-c con 0x7fdacc0a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:28.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.119+0000 7fdad39cf640 10 monclient(hunting): picked mon.noname-a con 0x7fdacc13f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:28.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.119+0000 7fdad39cf640 10 monclient(hunting): picked mon.noname-b con 0x7fdacc1430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:28.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.119+0000 7fdad39cf640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc1d7ba0 con 0x7fdacc13f710 2024-03-25T18:32:28.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.119+0000 7fdad39cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc1372f0 con 0x7fdacc1430c0 2024-03-25T18:32:28.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.119+0000 7fdad39cf640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc136820 con 0x7fdacc0a98f0 2024-03-25T18:32:28.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.119+0000 7fdad39cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:28.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.119+0000 7fdad39cf640 10 monclient(hunting): authenticate will time out at 7766.287136s 2024-03-25T18:32:28.137 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.135+0000 7fdad0f6a640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fdacc13f710 0x7fdacc1d7490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:42440/0 (socket says 172.21.15.99:42440) 2024-03-25T18:32:28.137 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.135+0000 7fdad0f6a640 1 -- 172.21.15.99:0/2323652550 learned_addr learned my addr 172.21.15.99:0/2323652550 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:28.137 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.136+0000 7fdac1ffb640 10 client.?.objecter ms_handle_connect 0x7fdacc13f710 2024-03-25T18:32:28.137 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.136+0000 7fdac1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:28.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.142+0000 7fdac1ffb640 10 client.?.objecter ms_handle_connect 0x7fdacc1430c0 2024-03-25T18:32:28.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.142+0000 7fdac1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:28.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.145+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256106413 0 0) 0x7fdacc1d7ba0 con 0x7fdacc13f710 2024-03-25T18:32:28.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.145+0000 7fdac1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:28.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.145+0000 7fdac1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:28.147 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.145+0000 7fdac1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:28.147 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.145+0000 7fdac1ffb640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:32:28.147 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.145+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab0004eb0 con 0x7fdacc13f710 2024-03-25T18:32:28.148 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.147+0000 7fdac1ffb640 10 client.?.objecter ms_handle_connect 0x7fdacc0a98f0 2024-03-25T18:32:28.148 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.147+0000 7fdac1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:28.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.154+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551901851 0 0) 0x7fdacc1372f0 con 0x7fdacc1430c0 2024-03-25T18:32:28.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.154+0000 7fdac1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:28.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.154+0000 7fdac1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:28.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.154+0000 7fdac1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:28.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.154+0000 7fdac1ffb640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:32:28.155 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.154+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdacc1d7ba0 con 0x7fdacc1430c0 2024-03-25T18:32:28.169 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.168+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2644531581 0 0) 0x7fdacc1d7ba0 con 0x7fdacc1430c0 2024-03-25T18:32:28.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.168+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdacc1372f0 con 0x7fdacc1430c0 2024-03-25T18:32:28.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.168+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fdac4001b10 con 0x7fdacc1430c0 2024-03-25T18:32:28.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.168+0000 7fdac1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:28.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.168+0000 7fdac1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:28.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.168+0000 7fdac1ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:28.170 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:28.170 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:28.170 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:28.170 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:28.170 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:28.170 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:28.171 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:28.171 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:28.171 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:28.171 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:28.171 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.168+0000 7fdac1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:28.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.172+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2045368884 0 0) 0x7fdab0004eb0 con 0x7fdacc13f710 2024-03-25T18:32:28.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.172+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdacc1d7ba0 con 0x7fdacc13f710 2024-03-25T18:32:28.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.172+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fdabc001a10 con 0x7fdacc13f710 2024-03-25T18:32:28.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.172+0000 7fdac1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:28.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.172+0000 7fdac1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:28.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.172+0000 7fdac1ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:28.174 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:28.174 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:28.174 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:28.174 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:28.174 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:28.174 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:28.174 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:28.174 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:28.174 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:28.174 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:28.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2209360353 0 0) 0x7fdacc1372f0 con 0x7fdacc1430c0 2024-03-25T18:32:28.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 >> v1:172.21.15.99:6790/0 conn(0x7fdacc0a98f0 legacy=0x7fdacc0b6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:28.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 >> v1:172.21.15.99:6789/0 conn(0x7fdacc13f710 legacy=0x7fdacc1d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:28.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdac1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:32:28.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdac1ffb640 1 monclient: found mon.b 2024-03-25T18:32:28.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdac1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:28.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdac1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:28.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc1369b0 con 0x7fdacc1430c0 2024-03-25T18:32:28.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdac1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:32:28.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdac1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:32:28.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdac1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:28.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdad39cf640 5 monclient: authenticate success, global_id 4546 2024-03-25T18:32:28.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdad39cf640 10 monclient: _renew_subs 2024-03-25T18:32:28.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdad39cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:28.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdad39cf640 1 -- 172.21.15.99:0/2323652550 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdacc0b13f0 con 0x7fdacc1430c0 2024-03-25T18:32:28.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdad39cf640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:28.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdad39cf640 10 monclient: _renew_subs 2024-03-25T18:32:28.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdad39cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:28.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.182+0000 7fdad39cf640 1 -- 172.21.15.99:0/2323652550 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdacc139780 con 0x7fdacc1430c0 2024-03-25T18:32:28.185 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.183+0000 7fdad39cf640 1 librados: init done 2024-03-25T18:32:28.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.183+0000 7fdad39cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:28.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.183+0000 7fdad39cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:28.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.183+0000 7fdad39cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:28.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.183+0000 7fdad39cf640 1 -- 172.21.15.99:0/2323652550 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdacc137480 con 0x7fdacc1430c0 2024-03-25T18:32:28.460 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.458+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdac4002000 con 0x7fdacc1430c0 2024-03-25T18:32:28.460 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.458+0000 7fdac1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:28.460 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.458+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fdac4003b30 con 0x7fdacc1430c0 2024-03-25T18:32:28.460 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.458+0000 7fdac1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:28.460 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.458+0000 7fdac1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:28.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.458+0000 7fdac1ffb640 10 monclient: dump: 2024-03-25T18:32:28.461 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:28.461 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:28.461 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:28.461 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:28.461 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:28.461 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:28.461 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:28.461 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:28.462 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:28.462 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:28.462 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.458+0000 7fdac37fe640 4 set_mon_vals no callback set 2024-03-25T18:32:28.462 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.459+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fdac4003d50 con 0x7fdacc1430c0 2024-03-25T18:32:28.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.459+0000 7fdad176b640 10 monclient: get_auth_request con 0x7fdab006de90 auth_method 0 2024-03-25T18:32:28.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.460+0000 7fdac1ffb640 10 client.4546.objecter ms_handle_connect 0x7fdab006de90 2024-03-25T18:32:28.464 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.463+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fdac4053810 con 0x7fdacc1430c0 2024-03-25T18:32:28.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.463+0000 7fdac1ffb640 10 client.4546.objecter ms_dispatch 0x7fdacc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:28.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.463+0000 7fdac1ffb640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:28.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.463+0000 7fdac1ffb640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:28.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.463+0000 7fdac1ffb640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T18:32:28.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.464+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdac404f320 con 0x7fdacc1430c0 2024-03-25T18:32:28.466 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.464+0000 7fdac1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:28.466 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.464+0000 7fdac1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:28.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.713+0000 7fdad39cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:28.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.713+0000 7fdad39cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:28.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.713+0000 7fdad39cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:28.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.713+0000 7fdad39cf640 1 -- 172.21.15.99:0/2323652550 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdacc0627b0 con 0x7fdacc1430c0 2024-03-25T18:32:28.734 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.733+0000 7fdac1ffb640 1 -- 172.21.15.99:0/2323652550 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fdac4051250 con 0x7fdacc1430c0 2024-03-25T18:32:28.734 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.733+0000 7fdac1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:28.734 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.733+0000 7fdac1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:28.734 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:32:28.734 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:28.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.735+0000 7fdad39cf640 10 librados: watch_flush enter 2024-03-25T18:32:28.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.735+0000 7fdad39cf640 10 librados: watch_flush exit 2024-03-25T18:32:28.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.735+0000 7fdad39cf640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T18:32:28.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.735+0000 7fdad39cf640 10 client.4546.objecter successfully canceled tick 2024-03-25T18:32:28.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.735+0000 7fdad39cf640 1 -- 172.21.15.99:0/2323652550 >> v1:172.21.15.190:6800/4220409699 conn(0x7fdab006de90 legacy=0x7fdab0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:28.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.735+0000 7fdad39cf640 10 monclient: shutdown 2024-03-25T18:32:28.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.735+0000 7fdad39cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:28.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.735+0000 7fdad39cf640 1 -- 172.21.15.99:0/2323652550 >> v1:172.21.15.190:6789/0 conn(0x7fdacc1430c0 legacy=0x7fdacc0b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:28.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.735+0000 7fdad39cf640 1 -- 172.21.15.99:0/2323652550 shutdown_connections 2024-03-25T18:32:28.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.735+0000 7fdad39cf640 1 -- 172.21.15.99:0/2323652550 >> 172.21.15.99:0/2323652550 conn(0x7fdacc0b7ea0 msgr2=0x7fdacc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:28.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.735+0000 7fdad39cf640 1 -- 172.21.15.99:0/2323652550 shutdown_connections 2024-03-25T18:32:28.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.735+0000 7fdad39cf640 1 -- 172.21.15.99:0/2323652550 wait complete. 2024-03-25T18:32:28.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:28.735+0000 7fdad39cf640 1 librados: shutdown 2024-03-25T18:32:28.749 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:29.604 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:07:30.197 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:29.648 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:07:30.207 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:29.650 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:07:30.209 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:29.656 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:07:30.248 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:29.659 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:07:30.251 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:29.659 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:07:30.251 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:29.688 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:07:30.246 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:29.691 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:07:30.283 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:29.697 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:07:30.289 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:35.555 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:35.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.666+0000 7fdff4c2a640 10 monclient: get_monmap_and_config 2024-03-25T18:32:35.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.666+0000 7fdff4c2a640 10 monclient: build_initial_monmap 2024-03-25T18:32:35.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.666+0000 7fdff4c2a640 10 monclient: monmap: 2024-03-25T18:32:35.668 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:35.668 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:35.668 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:32:35.667971+0000 2024-03-25T18:32:35.668 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:32:35.667971+0000 2024-03-25T18:32:35.668 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:35.668 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:35.669 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:35.669 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:32:35.669 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:32:35.669 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:35.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.667+0000 7fdff4c2a640 1 Processor -- start 2024-03-25T18:32:35.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.667+0000 7fdff4c2a640 1 -- start start 2024-03-25T18:32:35.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.667+0000 7fdff4c2a640 10 monclient: init 2024-03-25T18:32:35.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.668+0000 7fdff4c2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:35.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.668+0000 7fdff4c2a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:35.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.668+0000 7fdff4c2a640 10 monclient(hunting): picked mon.noname-a con 0x7fdff00a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:35.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.668+0000 7fdff4c2a640 10 monclient(hunting): picked mon.noname-b con 0x7fdff013f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:35.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.668+0000 7fdff4c2a640 10 monclient(hunting): picked mon.noname-c con 0x7fdff01430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:35.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.668+0000 7fdff4c2a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff0142520 con 0x7fdff00a98f0 2024-03-25T18:32:35.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.668+0000 7fdff4c2a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff00b28a0 con 0x7fdff01430c0 2024-03-25T18:32:35.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.668+0000 7fdff4c2a640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff00b2a30 con 0x7fdff013f710 2024-03-25T18:32:35.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.668+0000 7fdff4c2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:35.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.668+0000 7fdff4c2a640 10 monclient(hunting): authenticate will time out at 7773.836602s 2024-03-25T18:32:35.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.673+0000 7fdfedd9b640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7fdff013f710 0x7fdff01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:33420/0 (socket says 172.21.15.99:33420) 2024-03-25T18:32:35.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.673+0000 7fdfedd9b640 1 -- 172.21.15.99:0/4013631157 learned_addr learned my addr 172.21.15.99:0/4013631157 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:35.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.679+0000 7fdfed59a640 1 -- 172.21.15.99:0/4013631157 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722618834 0 0) 0x7fdff00b2a30 con 0x7fdff013f710 2024-03-25T18:32:35.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.679+0000 7fdfed59a640 10 monclient(hunting): _negotiate 2024-03-25T18:32:35.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.679+0000 7fdfed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:35.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.679+0000 7fdfed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:35.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.679+0000 7fdfed59a640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:32:35.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.679+0000 7fdfed59a640 1 -- 172.21.15.99:0/4013631157 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfd8003520 con 0x7fdff013f710 2024-03-25T18:32:35.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 1 -- 172.21.15.99:0/4013631157 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2146273919 0 0) 0x7fdfd8003520 con 0x7fdff013f710 2024-03-25T18:32:35.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 1 -- 172.21.15.99:0/4013631157 >> v1:172.21.15.190:6789/0 conn(0x7fdff01430c0 legacy=0x7fdff014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:35.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 1 -- 172.21.15.99:0/4013631157 >> v1:172.21.15.99:6789/0 conn(0x7fdff00a98f0 legacy=0x7fdff00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:35.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 10 monclient: _finish_hunting 0 2024-03-25T18:32:35.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 1 monclient: found mon.noname-b 2024-03-25T18:32:35.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:35.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:32:35.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 1 -- 172.21.15.99:0/4013631157 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff00b44c0 con 0x7fdff013f710 2024-03-25T18:32:35.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 10 monclient: _finish_auth 0 2024-03-25T18:32:35.690 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 10 monclient: _check_auth_tickets 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 1 -- 172.21.15.99:0/4013631157 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fdfe4001690 con 0x7fdff013f710 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdfed59a640 10 monclient: dump: 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:35.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdff4c2a640 5 monclient: authenticate success, global_id 4547 2024-03-25T18:32:35.692 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.688+0000 7fdff4c2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:35.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.699+0000 7fdfed59a640 1 -- 172.21.15.99:0/4013631157 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfe40017f0 con 0x7fdff013f710 2024-03-25T18:32:35.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.699+0000 7fdfed59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:35.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.699+0000 7fdfed59a640 1 -- 172.21.15.99:0/4013631157 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fdfe4001c00 con 0x7fdff013f710 2024-03-25T18:32:35.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.699+0000 7fdff4c2a640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:35.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.699+0000 7fdff4c2a640 4 set_mon_vals no callback set 2024-03-25T18:32:35.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.699+0000 7fdfed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:35.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.699+0000 7fdfed59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:35.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.699+0000 7fdfed59a640 10 monclient: dump: 2024-03-25T18:32:35.700 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:35.700 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:35.701 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:35.701 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:35.701 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:35.701 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:35.701 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:35.701 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:35.701 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:35.701 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:35.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.700+0000 7fdff4c2a640 10 monclient: shutdown 2024-03-25T18:32:35.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.700+0000 7fdff4c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:35.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.700+0000 7fdff4c2a640 1 -- 172.21.15.99:0/4013631157 >> v1:172.21.15.99:6790/0 conn(0x7fdff013f710 legacy=0x7fdff01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:35.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.700+0000 7fdff4c2a640 1 -- 172.21.15.99:0/4013631157 shutdown_connections 2024-03-25T18:32:35.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.700+0000 7fdff4c2a640 1 -- 172.21.15.99:0/4013631157 >> 172.21.15.99:0/4013631157 conn(0x7fdff00b7ea0 msgr2=0x7fdff00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:35.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.700+0000 7fdff4c2a640 1 -- 172.21.15.99:0/4013631157 shutdown_connections 2024-03-25T18:32:35.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.700+0000 7fdff4c2a640 1 -- 172.21.15.99:0/4013631157 wait complete. 2024-03-25T18:32:35.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.701+0000 7fdff4c2a640 10 monclient: build_initial_monmap 2024-03-25T18:32:35.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.701+0000 7fdff4c2a640 10 monclient: monmap: 2024-03-25T18:32:35.703 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:35.703 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:35.703 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:32:35.703 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:32:35.703 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:35.703 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:32:35.703 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:35.703 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:32:35.703 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:32:35.703 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:35.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.701+0000 7fdff4c2a640 1 librados: starting msgr at 2024-03-25T18:32:35.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.701+0000 7fdff4c2a640 1 librados: starting objecter 2024-03-25T18:32:35.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.701+0000 7fdff4c2a640 1 Processor -- start 2024-03-25T18:32:35.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.702+0000 7fdff4c2a640 1 -- start start 2024-03-25T18:32:35.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.702+0000 7fdff4c2a640 1 librados: setting wanted keys 2024-03-25T18:32:35.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.702+0000 7fdff4c2a640 1 librados: calling monclient init 2024-03-25T18:32:35.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.702+0000 7fdff4c2a640 10 monclient: init 2024-03-25T18:32:35.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.702+0000 7fdff4c2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:35.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.702+0000 7fdff4c2a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:35.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.702+0000 7fdff4c2a640 10 monclient(hunting): picked mon.noname-b con 0x7fdff00a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:35.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.702+0000 7fdff4c2a640 10 monclient(hunting): picked mon.noname-c con 0x7fdff013f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:35.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.703+0000 7fdff4c2a640 10 monclient(hunting): picked mon.noname-a con 0x7fdff01430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:35.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.703+0000 7fdff4c2a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff0138c10 con 0x7fdff01430c0 2024-03-25T18:32:35.705 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.703+0000 7fdff4c2a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff0138e60 con 0x7fdff00a98f0 2024-03-25T18:32:35.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.703+0000 7fdff4c2a640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff0139140 con 0x7fdff013f710 2024-03-25T18:32:35.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.703+0000 7fdff4c2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:35.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.703+0000 7fdff4c2a640 10 monclient(hunting): authenticate will time out at 7773.871038s 2024-03-25T18:32:35.710 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.709+0000 7fdfeed9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fdff01430c0 0x7fdff01e34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:55504/0 (socket says 172.21.15.99:55504) 2024-03-25T18:32:35.710 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.709+0000 7fdfeed9d640 1 -- 172.21.15.99:0/526886083 learned_addr learned my addr 172.21.15.99:0/526886083 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:35.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.713+0000 7fdfdeffd640 10 client.?.objecter ms_handle_connect 0x7fdff00a98f0 2024-03-25T18:32:35.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.713+0000 7fdfdeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:35.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.714+0000 7fdfdeffd640 10 client.?.objecter ms_handle_connect 0x7fdff01430c0 2024-03-25T18:32:35.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.714+0000 7fdfdeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:35.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.718+0000 7fdfdeffd640 10 client.?.objecter ms_handle_connect 0x7fdff013f710 2024-03-25T18:32:35.719 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.718+0000 7fdfdeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:35.720 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.719+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027740271 0 0) 0x7fdff0138e60 con 0x7fdff00a98f0 2024-03-25T18:32:35.720 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.719+0000 7fdfdeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:35.720 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.719+0000 7fdfdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:35.720 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.719+0000 7fdfdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:35.720 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.719+0000 7fdfdeffd640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:32:35.720 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.719+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfd0004eb0 con 0x7fdff00a98f0 2024-03-25T18:32:35.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.724+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209847307 0 0) 0x7fdff0138c10 con 0x7fdff01430c0 2024-03-25T18:32:35.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.724+0000 7fdfdeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:35.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.724+0000 7fdfdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:35.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.724+0000 7fdfdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:35.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.724+0000 7fdfdeffd640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:32:35.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.724+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff0138e60 con 0x7fdff01430c0 2024-03-25T18:32:35.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.738+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4036977958 0 0) 0x7fdfd0004eb0 con 0x7fdff00a98f0 2024-03-25T18:32:35.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.738+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdff0138c10 con 0x7fdff00a98f0 2024-03-25T18:32:35.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.738+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fdfe00018c0 con 0x7fdff00a98f0 2024-03-25T18:32:35.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.738+0000 7fdfdeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:35.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.738+0000 7fdfdeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:35.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.738+0000 7fdfdeffd640 10 monclient(hunting): dump: 2024-03-25T18:32:35.739 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:35.739 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:35.739 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:35.739 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:35.739 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:35.740 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:35.740 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:35.740 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:35.740 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:35.740 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:35.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.738+0000 7fdfdeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:35.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 120849003 0 0) 0x7fdff0138c10 con 0x7fdff00a98f0 2024-03-25T18:32:35.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 >> v1:172.21.15.99:6790/0 conn(0x7fdff013f710 legacy=0x7fdff0132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:35.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 >> v1:172.21.15.99:6789/0 conn(0x7fdff01430c0 legacy=0x7fdff01e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:35.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdfdeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:35.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdfdeffd640 1 monclient: found mon.b 2024-03-25T18:32:35.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdfdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:35.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdfdeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:35.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff0139920 con 0x7fdff00a98f0 2024-03-25T18:32:35.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdfdeffd640 10 monclient: _finish_auth 0 2024-03-25T18:32:35.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdfdeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:35.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdfdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:35.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdff4c2a640 5 monclient: authenticate success, global_id 4552 2024-03-25T18:32:35.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdff4c2a640 10 monclient: _renew_subs 2024-03-25T18:32:35.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdff4c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:35.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdff4c2a640 1 -- 172.21.15.99:0/526886083 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff01382e0 con 0x7fdff00a98f0 2024-03-25T18:32:35.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdff4c2a640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:35.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdff4c2a640 10 monclient: _renew_subs 2024-03-25T18:32:35.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdff4c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:35.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdff4c2a640 1 -- 172.21.15.99:0/526886083 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff00ab8b0 con 0x7fdff00a98f0 2024-03-25T18:32:35.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.743+0000 7fdff4c2a640 1 librados: init done 2024-03-25T18:32:35.746 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.744+0000 7fdff4c2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:35.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.744+0000 7fdff4c2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:35.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.744+0000 7fdff4c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:35.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:35.744+0000 7fdff4c2a640 1 -- 172.21.15.99:0/526886083 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff0137a20 con 0x7fdff00a98f0 2024-03-25T18:32:36.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.029+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfe00035f0 con 0x7fdff00a98f0 2024-03-25T18:32:36.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.029+0000 7fdfdeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:36.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.029+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fdfe0003a00 con 0x7fdff00a98f0 2024-03-25T18:32:36.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.029+0000 7fdfdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:36.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.029+0000 7fdfdeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:36.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.029+0000 7fdfdeffd640 10 monclient: dump: 2024-03-25T18:32:36.031 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:36.031 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:36.031 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:36.031 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:36.031 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:36.031 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:36.031 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:36.031 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:36.031 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:36.031 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:36.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.029+0000 7fdfecd99640 4 set_mon_vals no callback set 2024-03-25T18:32:36.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.030+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fdfe0003c20 con 0x7fdff00a98f0 2024-03-25T18:32:36.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.031+0000 7fdfedd9b640 10 monclient: get_auth_request con 0x7fdfd006dd10 auth_method 0 2024-03-25T18:32:36.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.031+0000 7fdfdeffd640 10 client.4552.objecter ms_handle_connect 0x7fdfd006dd10 2024-03-25T18:32:36.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.034+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fdfe00535b0 con 0x7fdff00a98f0 2024-03-25T18:32:36.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.034+0000 7fdfdeffd640 10 client.4552.objecter ms_dispatch 0x7fdff0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:36.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.034+0000 7fdfdeffd640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:36.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.034+0000 7fdfdeffd640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:36.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.035+0000 7fdfdeffd640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-25T18:32:36.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.036+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdfe004f0c0 con 0x7fdff00a98f0 2024-03-25T18:32:36.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.036+0000 7fdfdeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:36.037 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.036+0000 7fdfdeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:36.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.250+0000 7fdff4c2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:36.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.250+0000 7fdff4c2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:36.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.250+0000 7fdff4c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:36.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.250+0000 7fdff4c2a640 1 -- 172.21.15.99:0/526886083 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdff013d010 con 0x7fdff00a98f0 2024-03-25T18:32:36.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.278+0000 7fdfdeffd640 1 -- 172.21.15.99:0/526886083 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fdfe0050ff0 con 0x7fdff00a98f0 2024-03-25T18:32:36.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.278+0000 7fdfdeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:36.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.278+0000 7fdfdeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:36.279 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:32:36.279 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:36.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.280+0000 7fdff4c2a640 10 librados: watch_flush enter 2024-03-25T18:32:36.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.280+0000 7fdff4c2a640 10 librados: watch_flush exit 2024-03-25T18:32:36.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.280+0000 7fdff4c2a640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-25T18:32:36.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.280+0000 7fdff4c2a640 10 client.4552.objecter successfully canceled tick 2024-03-25T18:32:36.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.280+0000 7fdff4c2a640 1 -- 172.21.15.99:0/526886083 >> v1:172.21.15.190:6800/4220409699 conn(0x7fdfd006dd10 legacy=0x7fdfd00701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:36.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.280+0000 7fdff4c2a640 10 monclient: shutdown 2024-03-25T18:32:36.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.280+0000 7fdff4c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:36.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.280+0000 7fdff4c2a640 1 -- 172.21.15.99:0/526886083 >> v1:172.21.15.190:6789/0 conn(0x7fdff00a98f0 legacy=0x7fdff0137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:36.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.280+0000 7fdff4c2a640 1 -- 172.21.15.99:0/526886083 shutdown_connections 2024-03-25T18:32:36.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.280+0000 7fdff4c2a640 1 -- 172.21.15.99:0/526886083 >> 172.21.15.99:0/526886083 conn(0x7fdff00b7ea0 msgr2=0x7fdff0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:36.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.281+0000 7fdff4c2a640 1 -- 172.21.15.99:0/526886083 shutdown_connections 2024-03-25T18:32:36.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.281+0000 7fdff4c2a640 1 -- 172.21.15.99:0/526886083 wait complete. 2024-03-25T18:32:36.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:36.281+0000 7fdff4c2a640 1 librados: shutdown 2024-03-25T18:32:36.294 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:43.101 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:43.220 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.219+0000 7fdd57afb640 10 monclient: get_monmap_and_config 2024-03-25T18:32:43.220 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.219+0000 7fdd57afb640 10 monclient: build_initial_monmap 2024-03-25T18:32:43.220 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.219+0000 7fdd57afb640 10 monclient: monmap: 2024-03-25T18:32:43.220 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:43.220 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:43.220 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:32:43.220224+0000 2024-03-25T18:32:43.220 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:32:43.220224+0000 2024-03-25T18:32:43.221 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:43.221 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:43.221 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:43.221 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:32:43.221 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:32:43.221 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:43.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.219+0000 7fdd57afb640 1 Processor -- start 2024-03-25T18:32:43.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.220+0000 7fdd57afb640 1 -- start start 2024-03-25T18:32:43.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.220+0000 7fdd57afb640 10 monclient: init 2024-03-25T18:32:43.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.221+0000 7fdd57afb640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:43.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.221+0000 7fdd57afb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:43.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.221+0000 7fdd57afb640 10 monclient(hunting): picked mon.noname-c con 0x7fdd500b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:43.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.221+0000 7fdd57afb640 10 monclient(hunting): picked mon.noname-b con 0x7fdd500b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:43.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.221+0000 7fdd57afb640 10 monclient(hunting): picked mon.noname-a con 0x7fdd5014be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:43.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.221+0000 7fdd57afb640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd500af940 con 0x7fdd5014be90 2024-03-25T18:32:43.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.221+0000 7fdd57afb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd500b0a70 con 0x7fdd500b4d60 2024-03-25T18:32:43.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.221+0000 7fdd57afb640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd500b0d50 con 0x7fdd500b1730 2024-03-25T18:32:43.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.221+0000 7fdd57afb640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:43.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.221+0000 7fdd57afb640 10 monclient(hunting): authenticate will time out at 7781.389000s 2024-03-25T18:32:43.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.226+0000 7fdd55096640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7fdd500b1730 0x7fdd500b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:38312/0 (socket says 172.21.15.99:38312) 2024-03-25T18:32:43.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.226+0000 7fdd55096640 1 -- 172.21.15.99:0/1819232920 learned_addr learned my addr 172.21.15.99:0/1819232920 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:43.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.230+0000 7fdd54895640 1 -- 172.21.15.99:0/1819232920 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218897546 0 0) 0x7fdd500b0d50 con 0x7fdd500b1730 2024-03-25T18:32:43.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.230+0000 7fdd54895640 10 monclient(hunting): _negotiate 2024-03-25T18:32:43.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.230+0000 7fdd54895640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:43.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.230+0000 7fdd54895640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:43.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.230+0000 7fdd54895640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:32:43.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.230+0000 7fdd54895640 1 -- 172.21.15.99:0/1819232920 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd3c003520 con 0x7fdd500b1730 2024-03-25T18:32:43.237 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 1 -- 172.21.15.99:0/1819232920 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1087318814 0 0) 0x7fdd3c003520 con 0x7fdd500b1730 2024-03-25T18:32:43.237 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 1 -- 172.21.15.99:0/1819232920 >> v1:172.21.15.190:6789/0 conn(0x7fdd500b4d60 legacy=0x7fdd5013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:32:43.237 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 1 -- 172.21.15.99:0/1819232920 >> v1:172.21.15.99:6789/0 conn(0x7fdd5014be90 legacy=0x7fdd5014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:43.237 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 10 monclient: _finish_hunting 0 2024-03-25T18:32:43.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 1 monclient: found mon.noname-b 2024-03-25T18:32:43.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:43.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:32:43.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 1 -- 172.21.15.99:0/1819232920 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd500b1030 con 0x7fdd500b1730 2024-03-25T18:32:43.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 10 monclient: _finish_auth 0 2024-03-25T18:32:43.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 10 monclient: _check_auth_tickets 2024-03-25T18:32:43.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:43.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 1 -- 172.21.15.99:0/1819232920 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fdd40001690 con 0x7fdd500b1730 2024-03-25T18:32:43.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:43.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:43.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd54895640 10 monclient: dump: 2024-03-25T18:32:43.238 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:43.238 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:43.239 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:43.239 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:43.239 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:43.239 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:43.239 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:43.239 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:43.239 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:43.239 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:43.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd57afb640 5 monclient: authenticate success, global_id 4553 2024-03-25T18:32:43.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.235+0000 7fdd57afb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:43.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.252+0000 7fdd54895640 1 -- 172.21.15.99:0/1819232920 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd40001860 con 0x7fdd500b1730 2024-03-25T18:32:43.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.252+0000 7fdd54895640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:43.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.252+0000 7fdd54895640 1 -- 172.21.15.99:0/1819232920 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fdd40001cb0 con 0x7fdd500b1730 2024-03-25T18:32:43.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.252+0000 7fdd57afb640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:43.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.252+0000 7fdd57afb640 4 set_mon_vals no callback set 2024-03-25T18:32:43.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.252+0000 7fdd54895640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:43.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.252+0000 7fdd54895640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:43.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.252+0000 7fdd54895640 10 monclient: dump: 2024-03-25T18:32:43.255 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:43.255 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:43.255 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:43.255 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:43.256 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:43.256 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:43.256 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:43.256 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:43.256 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:43.256 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:43.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.253+0000 7fdd57afb640 10 monclient: shutdown 2024-03-25T18:32:43.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.253+0000 7fdd57afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:43.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.253+0000 7fdd57afb640 1 -- 172.21.15.99:0/1819232920 >> v1:172.21.15.99:6790/0 conn(0x7fdd500b1730 legacy=0x7fdd500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:43.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.253+0000 7fdd57afb640 1 -- 172.21.15.99:0/1819232920 shutdown_connections 2024-03-25T18:32:43.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.253+0000 7fdd57afb640 1 -- 172.21.15.99:0/1819232920 >> 172.21.15.99:0/1819232920 conn(0x7fdd500617a0 msgr2=0x7fdd50061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:43.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.253+0000 7fdd57afb640 1 -- 172.21.15.99:0/1819232920 shutdown_connections 2024-03-25T18:32:43.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.253+0000 7fdd57afb640 1 -- 172.21.15.99:0/1819232920 wait complete. 2024-03-25T18:32:43.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.254+0000 7fdd57afb640 10 monclient: build_initial_monmap 2024-03-25T18:32:43.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.254+0000 7fdd57afb640 10 monclient: monmap: 2024-03-25T18:32:43.259 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:43.260 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:43.260 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:32:43.260 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:32:43.260 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:43.260 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:32:43.260 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:43.260 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:32:43.260 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:32:43.260 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:43.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.254+0000 7fdd57afb640 1 librados: starting msgr at 2024-03-25T18:32:43.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.254+0000 7fdd57afb640 1 librados: starting objecter 2024-03-25T18:32:43.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.254+0000 7fdd57afb640 1 Processor -- start 2024-03-25T18:32:43.260 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.254+0000 7fdd57afb640 1 -- start start 2024-03-25T18:32:43.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.254+0000 7fdd57afb640 1 librados: setting wanted keys 2024-03-25T18:32:43.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.254+0000 7fdd57afb640 1 librados: calling monclient init 2024-03-25T18:32:43.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.254+0000 7fdd57afb640 10 monclient: init 2024-03-25T18:32:43.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.255+0000 7fdd57afb640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:43.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.255+0000 7fdd57afb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:43.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.255+0000 7fdd57afb640 10 monclient(hunting): picked mon.noname-b con 0x7fdd500b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:43.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.255+0000 7fdd57afb640 10 monclient(hunting): picked mon.noname-c con 0x7fdd500b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:43.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.255+0000 7fdd57afb640 10 monclient(hunting): picked mon.noname-a con 0x7fdd5014be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:43.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.255+0000 7fdd57afb640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd50142680 con 0x7fdd5014be90 2024-03-25T18:32:43.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.255+0000 7fdd57afb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5013bee0 con 0x7fdd500b1730 2024-03-25T18:32:43.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.255+0000 7fdd57afb640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5013b610 con 0x7fdd500b4d60 2024-03-25T18:32:43.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.255+0000 7fdd57afb640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:43.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.255+0000 7fdd57afb640 10 monclient(hunting): authenticate will time out at 7781.423594s 2024-03-25T18:32:43.262 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.261+0000 7fdd55096640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7fdd500b4d60 0x7fdd50142b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:38348/0 (socket says 172.21.15.99:38348) 2024-03-25T18:32:43.262 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.261+0000 7fdd55096640 1 -- 172.21.15.99:0/3001312717 learned_addr learned my addr 172.21.15.99:0/3001312717 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:43.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.266+0000 7fdd467fc640 10 client.?.objecter ms_handle_connect 0x7fdd500b4d60 2024-03-25T18:32:43.267 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.266+0000 7fdd467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:43.268 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.267+0000 7fdd467fc640 10 client.?.objecter ms_handle_connect 0x7fdd5014be90 2024-03-25T18:32:43.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.267+0000 7fdd467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:43.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.268+0000 7fdd467fc640 10 client.?.objecter ms_handle_connect 0x7fdd500b1730 2024-03-25T18:32:43.269 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.268+0000 7fdd467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:43.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.274+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942979805 0 0) 0x7fdd5013bee0 con 0x7fdd500b1730 2024-03-25T18:32:43.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.274+0000 7fdd467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:43.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.274+0000 7fdd467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:43.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.274+0000 7fdd467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:43.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.274+0000 7fdd467fc640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:32:43.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.274+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd34004eb0 con 0x7fdd500b1730 2024-03-25T18:32:43.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.282+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955083949 0 0) 0x7fdd5013b610 con 0x7fdd500b4d60 2024-03-25T18:32:43.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.282+0000 7fdd467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:43.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.282+0000 7fdd467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:43.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.282+0000 7fdd467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:43.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.282+0000 7fdd467fc640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:32:43.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.282+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd5013bee0 con 0x7fdd500b4d60 2024-03-25T18:32:43.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.286+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045917164 0 0) 0x7fdd50142680 con 0x7fdd5014be90 2024-03-25T18:32:43.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.286+0000 7fdd467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:43.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.286+0000 7fdd467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:43.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.286+0000 7fdd467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:43.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.286+0000 7fdd467fc640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:32:43.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.286+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd5013b610 con 0x7fdd5014be90 2024-03-25T18:32:43.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.288+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 882228750 0 0) 0x7fdd34004eb0 con 0x7fdd500b1730 2024-03-25T18:32:43.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.288+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd50142680 con 0x7fdd500b1730 2024-03-25T18:32:43.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.288+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fdd4c001870 con 0x7fdd500b1730 2024-03-25T18:32:43.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.288+0000 7fdd467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:43.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.288+0000 7fdd467fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:43.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.288+0000 7fdd467fc640 10 monclient(hunting): dump: 2024-03-25T18:32:43.290 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:43.290 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:43.290 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:43.290 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:43.290 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:43.291 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:43.291 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:43.291 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:43.291 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:43.291 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:43.291 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.288+0000 7fdd467fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:43.298 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2482498979 0 0) 0x7fdd50142680 con 0x7fdd500b1730 2024-03-25T18:32:43.299 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 >> v1:172.21.15.99:6790/0 conn(0x7fdd500b4d60 legacy=0x7fdd50142b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:43.299 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 >> v1:172.21.15.99:6789/0 conn(0x7fdd5014be90 legacy=0x7fdd501e21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:43.299 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd467fc640 10 monclient: _finish_hunting 0 2024-03-25T18:32:43.299 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd467fc640 1 monclient: found mon.b 2024-03-25T18:32:43.299 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:43.299 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd467fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:43.299 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5013b7a0 con 0x7fdd500b1730 2024-03-25T18:32:43.299 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd467fc640 10 monclient: _finish_auth 0 2024-03-25T18:32:43.299 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd467fc640 10 monclient: _check_auth_tickets 2024-03-25T18:32:43.299 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:43.299 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd57afb640 5 monclient: authenticate success, global_id 4555 2024-03-25T18:32:43.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd57afb640 10 monclient: _renew_subs 2024-03-25T18:32:43.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd57afb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:43.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd57afb640 1 -- 172.21.15.99:0/3001312717 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd50140d00 con 0x7fdd500b1730 2024-03-25T18:32:43.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd57afb640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:43.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd57afb640 10 monclient: _renew_subs 2024-03-25T18:32:43.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd57afb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:43.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd57afb640 1 -- 172.21.15.99:0/3001312717 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd50143400 con 0x7fdd500b1730 2024-03-25T18:32:43.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.296+0000 7fdd57afb640 1 librados: init done 2024-03-25T18:32:43.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.297+0000 7fdd57afb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:43.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.297+0000 7fdd57afb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:43.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.297+0000 7fdd57afb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:43.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.297+0000 7fdd57afb640 1 -- 172.21.15.99:0/3001312717 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd500627b0 con 0x7fdd500b1730 2024-03-25T18:32:43.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.575+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd4c0035f0 con 0x7fdd500b1730 2024-03-25T18:32:43.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.575+0000 7fdd467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:43.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.575+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fdd4c001d80 con 0x7fdd500b1730 2024-03-25T18:32:43.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.575+0000 7fdd467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:43.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.576+0000 7fdd467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:43.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.576+0000 7fdd467fc640 10 monclient: dump: 2024-03-25T18:32:43.578 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:43.578 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:43.578 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:43.578 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:43.578 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:43.578 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:43.578 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:43.578 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:43.578 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:43.578 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:43.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.575+0000 7fdd47fff640 4 set_mon_vals no callback set 2024-03-25T18:32:43.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.577+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fdd4c0197f0 con 0x7fdd500b1730 2024-03-25T18:32:43.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.577+0000 7fdd55096640 10 monclient: get_auth_request con 0x7fdd3406dda0 auth_method 0 2024-03-25T18:32:43.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.577+0000 7fdd467fc640 10 client.4555.objecter ms_handle_connect 0x7fdd3406dda0 2024-03-25T18:32:43.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.581+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fdd4c053600 con 0x7fdd500b1730 2024-03-25T18:32:43.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.581+0000 7fdd467fc640 10 client.4555.objecter ms_dispatch 0x7fdd50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:43.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.581+0000 7fdd467fc640 3 client.4555.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:43.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.581+0000 7fdd467fc640 3 client.4555.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:43.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.581+0000 7fdd467fc640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-25T18:32:43.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.582+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd4c04ef40 con 0x7fdd500b1730 2024-03-25T18:32:43.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.582+0000 7fdd467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:43.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.582+0000 7fdd467fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:43.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.831+0000 7fdd57afb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:43.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.831+0000 7fdd57afb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:43.833 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.831+0000 7fdd57afb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:43.833 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.831+0000 7fdd57afb640 1 -- 172.21.15.99:0/3001312717 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd50069f40 con 0x7fdd500b1730 2024-03-25T18:32:43.850 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.849+0000 7fdd467fc640 1 -- 172.21.15.99:0/3001312717 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fdd4c051040 con 0x7fdd500b1730 2024-03-25T18:32:43.850 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.849+0000 7fdd467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:43.850 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.849+0000 7fdd467fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:43.850 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:32:43.850 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:43.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.851+0000 7fdd57afb640 10 librados: watch_flush enter 2024-03-25T18:32:43.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.851+0000 7fdd57afb640 10 librados: watch_flush exit 2024-03-25T18:32:43.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.851+0000 7fdd57afb640 20 client.4555.objecter shutdown clearing up homeless session... 2024-03-25T18:32:43.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.851+0000 7fdd57afb640 10 client.4555.objecter successfully canceled tick 2024-03-25T18:32:43.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.851+0000 7fdd57afb640 1 -- 172.21.15.99:0/3001312717 >> v1:172.21.15.190:6800/4220409699 conn(0x7fdd3406dda0 legacy=0x7fdd34070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:43.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.851+0000 7fdd57afb640 10 monclient: shutdown 2024-03-25T18:32:43.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.851+0000 7fdd57afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:43.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.851+0000 7fdd57afb640 1 -- 172.21.15.99:0/3001312717 >> v1:172.21.15.190:6789/0 conn(0x7fdd500b1730 legacy=0x7fdd501405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:43.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.851+0000 7fdd57afb640 1 -- 172.21.15.99:0/3001312717 shutdown_connections 2024-03-25T18:32:43.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.851+0000 7fdd57afb640 1 -- 172.21.15.99:0/3001312717 >> 172.21.15.99:0/3001312717 conn(0x7fdd500617a0 msgr2=0x7fdd500b6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:43.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.851+0000 7fdd57afb640 1 -- 172.21.15.99:0/3001312717 shutdown_connections 2024-03-25T18:32:43.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.851+0000 7fdd57afb640 1 -- 172.21.15.99:0/3001312717 wait complete. 2024-03-25T18:32:43.854 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:43.852+0000 7fdd57afb640 1 librados: shutdown 2024-03-25T18:32:43.865 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:49.716 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:07:50.308 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:49.719 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:07:50.311 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:49.743 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:07:50.336 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:49.768 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:07:50.327 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:49.778 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:07:50.337 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:49.791 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:07:50.383 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:49.803 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:07:50.395 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:49.808 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:07:50.366 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:49.825 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:07:50.418 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:50.671 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:50.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.782+0000 7f55c9674640 10 monclient: get_monmap_and_config 2024-03-25T18:32:50.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.782+0000 7f55c9674640 10 monclient: build_initial_monmap 2024-03-25T18:32:50.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.782+0000 7f55c9674640 10 monclient: monmap: 2024-03-25T18:32:50.784 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:50.784 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:50.784 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:32:50.783907+0000 2024-03-25T18:32:50.784 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:32:50.783907+0000 2024-03-25T18:32:50.784 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:50.784 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:50.784 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:50.785 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:32:50.785 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:32:50.785 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:50.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.783+0000 7f55c9674640 1 Processor -- start 2024-03-25T18:32:50.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.783+0000 7f55c9674640 1 -- start start 2024-03-25T18:32:50.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.783+0000 7f55c9674640 10 monclient: init 2024-03-25T18:32:50.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.784+0000 7f55c9674640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:50.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.784+0000 7f55c9674640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:50.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.784+0000 7f55c9674640 10 monclient(hunting): picked mon.noname-b con 0x7f55c40a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:50.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.784+0000 7f55c9674640 10 monclient(hunting): picked mon.noname-c con 0x7f55c413f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:50.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.784+0000 7f55c9674640 10 monclient(hunting): picked mon.noname-a con 0x7f55c41430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:50.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.784+0000 7f55c9674640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c4142520 con 0x7f55c41430c0 2024-03-25T18:32:50.787 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.784+0000 7f55c9674640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c40b28a0 con 0x7f55c413f710 2024-03-25T18:32:50.787 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.784+0000 7f55c9674640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c40b2a30 con 0x7f55c40a98f0 2024-03-25T18:32:50.787 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.784+0000 7f55c9674640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:50.787 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.784+0000 7f55c9674640 10 monclient(hunting): authenticate will time out at 7788.952202s 2024-03-25T18:32:50.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.793+0000 7f55c27fc640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f55c413f710 0x7f55c41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:37604/0 (socket says 172.21.15.99:37604) 2024-03-25T18:32:50.794 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.793+0000 7f55c27fc640 1 -- 172.21.15.99:0/38312745 learned_addr learned my addr 172.21.15.99:0/38312745 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:50.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.801+0000 7f55c1ffb640 1 -- 172.21.15.99:0/38312745 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342913962 0 0) 0x7f55c40b28a0 con 0x7f55c413f710 2024-03-25T18:32:50.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.801+0000 7f55c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:50.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.801+0000 7f55c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:50.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.801+0000 7f55c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:50.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.801+0000 7f55c1ffb640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:32:50.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.801+0000 7f55c1ffb640 1 -- 172.21.15.99:0/38312745 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55ac003520 con 0x7f55c413f710 2024-03-25T18:32:50.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.807+0000 7f55c1ffb640 1 -- 172.21.15.99:0/38312745 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365821776 0 0) 0x7f55c40b2a30 con 0x7f55c40a98f0 2024-03-25T18:32:50.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.807+0000 7f55c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:50.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.807+0000 7f55c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:50.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.807+0000 7f55c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:50.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.807+0000 7f55c1ffb640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:32:50.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.807+0000 7f55c1ffb640 1 -- 172.21.15.99:0/38312745 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c40b28a0 con 0x7f55c40a98f0 2024-03-25T18:32:50.810 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.809+0000 7f55c1ffb640 1 -- 172.21.15.99:0/38312745 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 469463024 0 0) 0x7f55ac003520 con 0x7f55c413f710 2024-03-25T18:32:50.810 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.809+0000 7f55c1ffb640 1 -- 172.21.15.99:0/38312745 >> v1:172.21.15.99:6790/0 conn(0x7f55c40a98f0 legacy=0x7f55c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:50.810 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.809+0000 7f55c1ffb640 1 -- 172.21.15.99:0/38312745 >> v1:172.21.15.99:6789/0 conn(0x7f55c41430c0 legacy=0x7f55c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:50.810 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.809+0000 7f55c1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:32:50.810 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.809+0000 7f55c1ffb640 1 monclient: found mon.noname-c 2024-03-25T18:32:50.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.809+0000 7f55c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:50.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.809+0000 7f55c1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:32:50.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.809+0000 7f55c1ffb640 1 -- 172.21.15.99:0/38312745 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c40b44c0 con 0x7f55c413f710 2024-03-25T18:32:50.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.809+0000 7f55c1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:32:50.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.809+0000 7f55c1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:32:50.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.809+0000 7f55c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:50.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.809+0000 7f55c9674640 5 monclient: authenticate success, global_id 4558 2024-03-25T18:32:50.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.809+0000 7f55c9674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:50.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.816+0000 7f55c1ffb640 1 -- 172.21.15.99:0/38312745 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f55b8001650 con 0x7f55c413f710 2024-03-25T18:32:50.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.816+0000 7f55c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:50.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:50.818 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c1ffb640 10 monclient: dump: 2024-03-25T18:32:50.818 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:50.818 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:50.818 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:50.818 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:50.818 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c1ffb640 1 -- 172.21.15.99:0/38312745 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55b80018a0 con 0x7f55c413f710 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c9674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c9674640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c1ffb640 1 -- 172.21.15.99:0/38312745 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f55b8001c30 con 0x7f55c413f710 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c9674640 4 set_mon_vals no callback set 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c1ffb640 10 monclient: dump: 2024-03-25T18:32:50.819 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:50.820 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:50.820 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:50.820 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:50.820 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:50.820 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:50.820 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:50.820 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:50.820 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:50.820 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:50.822 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c9674640 10 monclient: shutdown 2024-03-25T18:32:50.822 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c9674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:50.822 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c9674640 1 -- 172.21.15.99:0/38312745 >> v1:172.21.15.190:6789/0 conn(0x7f55c413f710 legacy=0x7f55c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:50.822 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c9674640 1 -- 172.21.15.99:0/38312745 shutdown_connections 2024-03-25T18:32:50.822 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c9674640 1 -- 172.21.15.99:0/38312745 >> 172.21.15.99:0/38312745 conn(0x7f55c40b7ea0 msgr2=0x7f55c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:50.822 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c9674640 1 -- 172.21.15.99:0/38312745 shutdown_connections 2024-03-25T18:32:50.822 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.817+0000 7f55c9674640 1 -- 172.21.15.99:0/38312745 wait complete. 2024-03-25T18:32:50.822 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.818+0000 7f55c9674640 10 monclient: build_initial_monmap 2024-03-25T18:32:50.822 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.818+0000 7f55c9674640 10 monclient: monmap: 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.818+0000 7f55c9674640 1 librados: starting msgr at 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.818+0000 7f55c9674640 1 librados: starting objecter 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 1 Processor -- start 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 1 -- start start 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 1 librados: setting wanted keys 2024-03-25T18:32:50.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 1 librados: calling monclient init 2024-03-25T18:32:50.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 10 monclient: init 2024-03-25T18:32:50.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:50.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:50.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 10 monclient(hunting): picked mon.noname-b con 0x7f55c40a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:50.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 10 monclient(hunting): picked mon.noname-a con 0x7f55c413f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:50.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 10 monclient(hunting): picked mon.noname-c con 0x7f55c41430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:50.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c41d4f60 con 0x7f55c413f710 2024-03-25T18:32:50.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c41d4b60 con 0x7f55c40a98f0 2024-03-25T18:32:50.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c41d4cf0 con 0x7f55c41430c0 2024-03-25T18:32:50.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:50.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.819+0000 7f55c9674640 10 monclient(hunting): authenticate will time out at 7788.987622s 2024-03-25T18:32:50.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.824+0000 7f55c37fe640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f55c41430c0 0x7f55c41e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:59378/0 (socket says 172.21.15.99:59378) 2024-03-25T18:32:50.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.824+0000 7f55c37fe640 1 -- 172.21.15.99:0/444597754 learned_addr learned my addr 172.21.15.99:0/444597754 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:50.827 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.826+0000 7f55ab7fe640 10 client.?.objecter ms_handle_connect 0x7f55c41430c0 2024-03-25T18:32:50.827 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.826+0000 7f55ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:50.836 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.835+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611593480 0 0) 0x7f55c41d4cf0 con 0x7f55c41430c0 2024-03-25T18:32:50.836 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.835+0000 7f55ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:50.836 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.835+0000 7f55ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:50.836 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.835+0000 7f55ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:50.836 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.835+0000 7f55ab7fe640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:32:50.836 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.835+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b0004eb0 con 0x7f55c41430c0 2024-03-25T18:32:50.838 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.837+0000 7f55ab7fe640 10 client.?.objecter ms_handle_connect 0x7f55c413f710 2024-03-25T18:32:50.838 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.837+0000 7f55ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:50.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.840+0000 7f55ab7fe640 10 client.?.objecter ms_handle_connect 0x7f55c40a98f0 2024-03-25T18:32:50.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.840+0000 7f55ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:50.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.847+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780935173 0 0) 0x7f55c41d4b60 con 0x7f55c40a98f0 2024-03-25T18:32:50.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.847+0000 7f55ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:50.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.847+0000 7f55ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:50.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.847+0000 7f55ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:50.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.847+0000 7f55ab7fe640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:32:50.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.847+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c41d4cf0 con 0x7f55c40a98f0 2024-03-25T18:32:50.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.847+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218301601 0 0) 0x7f55c41d4f60 con 0x7f55c413f710 2024-03-25T18:32:50.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.847+0000 7f55ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:50.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.847+0000 7f55ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:50.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.847+0000 7f55ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:50.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.847+0000 7f55ab7fe640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:32:50.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.847+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c41d4b60 con 0x7f55c413f710 2024-03-25T18:32:50.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.849+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4008066804 0 0) 0x7f55b0004eb0 con 0x7f55c41430c0 2024-03-25T18:32:50.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.849+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55c41d4f60 con 0x7f55c41430c0 2024-03-25T18:32:50.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.849+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f55bc001c30 con 0x7f55c41430c0 2024-03-25T18:32:50.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.849+0000 7f55ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:50.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.849+0000 7f55ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:50.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.849+0000 7f55ab7fe640 10 monclient(hunting): dump: 2024-03-25T18:32:50.851 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:50.851 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:50.851 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:50.851 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:50.851 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:50.851 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:50.852 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:50.852 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:50.852 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:50.852 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:50.852 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.849+0000 7f55ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:50.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 243839353 0 0) 0x7f55c41d4f60 con 0x7f55c41430c0 2024-03-25T18:32:50.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 >> v1:172.21.15.190:6789/0 conn(0x7f55c40a98f0 legacy=0x7f55c41d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:50.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 >> v1:172.21.15.99:6789/0 conn(0x7f55c413f710 legacy=0x7f55c41d76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:50.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:50.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55ab7fe640 1 monclient: found mon.c 2024-03-25T18:32:50.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:50.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55ab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:32:50.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c41d7de0 con 0x7f55c41430c0 2024-03-25T18:32:50.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55ab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:50.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:50.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:50.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55c9674640 5 monclient: authenticate success, global_id 4562 2024-03-25T18:32:50.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55c9674640 10 monclient: _renew_subs 2024-03-25T18:32:50.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55c9674640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:32:50.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55c9674640 1 -- 172.21.15.99:0/444597754 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c41d8090 con 0x7f55c41430c0 2024-03-25T18:32:50.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55c9674640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:50.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55c9674640 10 monclient: _renew_subs 2024-03-25T18:32:50.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55c9674640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:32:50.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55c9674640 1 -- 172.21.15.99:0/444597754 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c4138b40 con 0x7f55c41430c0 2024-03-25T18:32:50.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.860+0000 7f55c9674640 1 librados: init done 2024-03-25T18:32:50.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.861+0000 7f55c9674640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:50.864 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.861+0000 7f55c9674640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:50.864 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.861+0000 7f55c9674640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:32:50.864 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.861+0000 7f55c9674640 1 -- 172.21.15.99:0/444597754 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c4138e40 con 0x7f55c41430c0 2024-03-25T18:32:50.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.884+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55bc003970 con 0x7f55c41430c0 2024-03-25T18:32:50.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.884+0000 7f55ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:50.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.884+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f55bc003d00 con 0x7f55c41430c0 2024-03-25T18:32:50.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.884+0000 7f55ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:50.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.884+0000 7f55ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:50.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.884+0000 7f55ab7fe640 10 monclient: dump: 2024-03-25T18:32:50.886 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:50.886 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:50.886 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:50.886 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:50.886 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:50.886 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:50.886 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:50.886 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:50.886 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:50.886 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:50.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.884+0000 7f55c17fa640 4 set_mon_vals no callback set 2024-03-25T18:32:50.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.885+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f55bc003f20 con 0x7f55c41430c0 2024-03-25T18:32:50.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.886+0000 7f55c2ffd640 10 monclient: get_auth_request con 0x7f55b006ddf0 auth_method 0 2024-03-25T18:32:50.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:50.886+0000 7f55ab7fe640 10 client.4562.objecter ms_handle_connect 0x7f55b006ddf0 2024-03-25T18:32:51.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.159+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f55bc0549e0 con 0x7f55c41430c0 2024-03-25T18:32:51.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.159+0000 7f55ab7fe640 10 client.4562.objecter ms_dispatch 0x7f55c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:51.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.159+0000 7f55ab7fe640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:51.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.159+0000 7f55ab7fe640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:51.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.159+0000 7f55ab7fe640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-25T18:32:51.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.159+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55bc0504f0 con 0x7f55c41430c0 2024-03-25T18:32:51.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.159+0000 7f55ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:51.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.159+0000 7f55ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:51.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.402+0000 7f55c9674640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:51.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.402+0000 7f55c9674640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:51.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.402+0000 7f55c9674640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:32:51.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.402+0000 7f55c9674640 1 -- 172.21.15.99:0/444597754 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55c40627b0 con 0x7f55c41430c0 2024-03-25T18:32:51.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.428+0000 7f55ab7fe640 1 -- 172.21.15.99:0/444597754 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f55bc052420 con 0x7f55c41430c0 2024-03-25T18:32:51.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.428+0000 7f55ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:51.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.428+0000 7f55ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:51.430 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:32:51.430 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:51.432 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.431+0000 7f55c9674640 10 librados: watch_flush enter 2024-03-25T18:32:51.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.431+0000 7f55c9674640 10 librados: watch_flush exit 2024-03-25T18:32:51.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.431+0000 7f55c9674640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-25T18:32:51.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.431+0000 7f55c9674640 10 client.4562.objecter successfully canceled tick 2024-03-25T18:32:51.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.431+0000 7f55c9674640 1 -- 172.21.15.99:0/444597754 >> v1:172.21.15.190:6800/4220409699 conn(0x7f55b006ddf0 legacy=0x7f55b00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:51.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.431+0000 7f55c9674640 10 monclient: shutdown 2024-03-25T18:32:51.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.431+0000 7f55c9674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:51.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.431+0000 7f55c9674640 1 -- 172.21.15.99:0/444597754 >> v1:172.21.15.99:6790/0 conn(0x7f55c41430c0 legacy=0x7f55c41e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:51.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.431+0000 7f55c9674640 1 -- 172.21.15.99:0/444597754 shutdown_connections 2024-03-25T18:32:51.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.431+0000 7f55c9674640 1 -- 172.21.15.99:0/444597754 >> 172.21.15.99:0/444597754 conn(0x7f55c40b7ea0 msgr2=0x7f55c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:51.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.431+0000 7f55c9674640 1 -- 172.21.15.99:0/444597754 shutdown_connections 2024-03-25T18:32:51.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.431+0000 7f55c9674640 1 -- 172.21.15.99:0/444597754 wait complete. 2024-03-25T18:32:51.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:51.431+0000 7f55c9674640 1 librados: shutdown 2024-03-25T18:32:51.444 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:58.251 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:58.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.364+0000 7f014dc3e640 10 monclient: get_monmap_and_config 2024-03-25T18:32:58.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.364+0000 7f014dc3e640 10 monclient: build_initial_monmap 2024-03-25T18:32:58.366 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.364+0000 7f014dc3e640 10 monclient: monmap: 2024-03-25T18:32:58.366 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:58.366 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:58.366 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:32:58.365657+0000 2024-03-25T18:32:58.366 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:32:58.365657+0000 2024-03-25T18:32:58.366 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:58.366 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:58.366 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:58.366 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:32:58.366 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:32:58.366 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:58.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.365+0000 7f014dc3e640 1 Processor -- start 2024-03-25T18:32:58.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.365+0000 7f014dc3e640 1 -- start start 2024-03-25T18:32:58.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.365+0000 7f014dc3e640 10 monclient: init 2024-03-25T18:32:58.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.366+0000 7f014dc3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:58.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.366+0000 7f014dc3e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:58.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.366+0000 7f014dc3e640 10 monclient(hunting): picked mon.noname-a con 0x7f01480a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:58.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.366+0000 7f014dc3e640 10 monclient(hunting): picked mon.noname-c con 0x7f014813f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:58.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.366+0000 7f014dc3e640 10 monclient(hunting): picked mon.noname-b con 0x7f01481430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:58.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.366+0000 7f014dc3e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0148142520 con 0x7f01480a98f0 2024-03-25T18:32:58.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.366+0000 7f014dc3e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01480b28a0 con 0x7f014813f710 2024-03-25T18:32:58.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.366+0000 7f014dc3e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01480b2a30 con 0x7f01481430c0 2024-03-25T18:32:58.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.366+0000 7f014dc3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:58.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.366+0000 7f014dc3e640 10 monclient(hunting): authenticate will time out at 7796.533996s 2024-03-25T18:32:58.378 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.377+0000 7f0147fff640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f01481430c0 0x7f014814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:49304/0 (socket says 172.21.15.99:49304) 2024-03-25T18:32:58.378 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.377+0000 7f0147fff640 1 -- 172.21.15.99:0/1069470859 learned_addr learned my addr 172.21.15.99:0/1069470859 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:58.390 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.389+0000 7f01467fc640 1 -- 172.21.15.99:0/1069470859 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846067811 0 0) 0x7f01480b28a0 con 0x7f014813f710 2024-03-25T18:32:58.391 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.389+0000 7f01467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:58.391 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.389+0000 7f01467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:58.391 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.389+0000 7f01467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:58.391 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.389+0000 7f01467fc640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:32:58.391 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.389+0000 7f01467fc640 1 -- 172.21.15.99:0/1069470859 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0130003520 con 0x7f014813f710 2024-03-25T18:32:58.391 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.389+0000 7f01467fc640 1 -- 172.21.15.99:0/1069470859 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244965264 0 0) 0x7f01480b2a30 con 0x7f01481430c0 2024-03-25T18:32:58.391 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.389+0000 7f01467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:58.391 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.389+0000 7f01467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:58.391 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.389+0000 7f01467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:58.391 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.389+0000 7f01467fc640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:32:58.391 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.389+0000 7f01467fc640 1 -- 172.21.15.99:0/1069470859 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01480b28a0 con 0x7f01481430c0 2024-03-25T18:32:58.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 1 -- 172.21.15.99:0/1069470859 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 153993503 0 0) 0x7f0130003520 con 0x7f014813f710 2024-03-25T18:32:58.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 1 -- 172.21.15.99:0/1069470859 >> v1:172.21.15.99:6790/0 conn(0x7f01481430c0 legacy=0x7f014814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:58.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 1 -- 172.21.15.99:0/1069470859 >> v1:172.21.15.99:6789/0 conn(0x7f01480a98f0 legacy=0x7f01480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:58.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 10 monclient: _finish_hunting 0 2024-03-25T18:32:58.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 1 monclient: found mon.noname-c 2024-03-25T18:32:58.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:58.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:32:58.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 1 -- 172.21.15.99:0/1069470859 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01480b44c0 con 0x7f014813f710 2024-03-25T18:32:58.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 10 monclient: _finish_auth 0 2024-03-25T18:32:58.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 10 monclient: _check_auth_tickets 2024-03-25T18:32:58.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:58.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 1 -- 172.21.15.99:0/1069470859 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f013c001690 con 0x7f014813f710 2024-03-25T18:32:58.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:58.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:58.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f01467fc640 10 monclient: dump: 2024-03-25T18:32:58.400 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:58.400 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:58.400 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:58.400 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:58.400 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:58.400 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:58.400 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:58.400 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:58.400 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:58.400 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:58.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f014dc3e640 5 monclient: authenticate success, global_id 4564 2024-03-25T18:32:58.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.397+0000 7f014dc3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:58.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.402+0000 7f01467fc640 1 -- 172.21.15.99:0/1069470859 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f013c0017f0 con 0x7f014813f710 2024-03-25T18:32:58.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.402+0000 7f01467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:58.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.402+0000 7f01467fc640 1 -- 172.21.15.99:0/1069470859 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f013c001c00 con 0x7f014813f710 2024-03-25T18:32:58.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.402+0000 7f014dc3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:58.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.402+0000 7f014dc3e640 4 set_mon_vals no callback set 2024-03-25T18:32:58.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.402+0000 7f01467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:58.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.402+0000 7f01467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:58.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.402+0000 7f01467fc640 10 monclient: dump: 2024-03-25T18:32:58.405 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:58.405 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:58.405 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:58.405 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:58.405 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:58.405 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:58.405 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:58.405 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:58.405 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:58.405 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:58.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.403+0000 7f014dc3e640 10 monclient: shutdown 2024-03-25T18:32:58.405 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.403+0000 7f014dc3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:58.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.403+0000 7f014dc3e640 1 -- 172.21.15.99:0/1069470859 >> v1:172.21.15.190:6789/0 conn(0x7f014813f710 legacy=0x7f01481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:58.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.403+0000 7f014dc3e640 1 -- 172.21.15.99:0/1069470859 shutdown_connections 2024-03-25T18:32:58.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.403+0000 7f014dc3e640 1 -- 172.21.15.99:0/1069470859 >> 172.21.15.99:0/1069470859 conn(0x7f01480b7ea0 msgr2=0x7f01480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:58.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.403+0000 7f014dc3e640 1 -- 172.21.15.99:0/1069470859 shutdown_connections 2024-03-25T18:32:58.406 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.403+0000 7f014dc3e640 1 -- 172.21.15.99:0/1069470859 wait complete. 2024-03-25T18:32:58.408 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.404+0000 7f014dc3e640 10 monclient: build_initial_monmap 2024-03-25T18:32:58.408 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.404+0000 7f014dc3e640 10 monclient: monmap: 2024-03-25T18:32:58.408 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:32:58.408 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:58.408 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:32:58.408 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:32:58.408 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:58.408 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:32:58.408 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:32:58.408 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:32:58.408 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:32:58.408 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:58.408 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.404+0000 7f014dc3e640 1 librados: starting msgr at 2024-03-25T18:32:58.408 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.404+0000 7f014dc3e640 1 librados: starting objecter 2024-03-25T18:32:58.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.404+0000 7f014dc3e640 1 Processor -- start 2024-03-25T18:32:58.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.405+0000 7f014dc3e640 1 -- start start 2024-03-25T18:32:58.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.405+0000 7f014dc3e640 1 librados: setting wanted keys 2024-03-25T18:32:58.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.405+0000 7f014dc3e640 1 librados: calling monclient init 2024-03-25T18:32:58.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.405+0000 7f014dc3e640 10 monclient: init 2024-03-25T18:32:58.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.405+0000 7f014dc3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:58.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.405+0000 7f014dc3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:58.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.405+0000 7f014dc3e640 10 monclient(hunting): picked mon.noname-c con 0x7f01480a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:32:58.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.405+0000 7f014dc3e640 10 monclient(hunting): picked mon.noname-b con 0x7f014813f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:32:58.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.406+0000 7f014dc3e640 10 monclient(hunting): picked mon.noname-a con 0x7f01481430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:32:58.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.406+0000 7f014dc3e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0148139070 con 0x7f01481430c0 2024-03-25T18:32:58.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.406+0000 7f014dc3e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0148133710 con 0x7f014813f710 2024-03-25T18:32:58.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.406+0000 7f014dc3e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01481338a0 con 0x7f01480a98f0 2024-03-25T18:32:58.409 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.406+0000 7f014dc3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:58.410 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.406+0000 7f014dc3e640 10 monclient(hunting): authenticate will time out at 7796.574075s 2024-03-25T18:32:58.422 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.421+0000 7f0146ffd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f014813f710 0x7f01481daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:39362/0 (socket says 172.21.15.99:39362) 2024-03-25T18:32:58.422 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.421+0000 7f0146ffd640 1 -- 172.21.15.99:0/1034407030 learned_addr learned my addr 172.21.15.99:0/1034407030 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:32:58.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.423+0000 7f012ffff640 10 client.?.objecter ms_handle_connect 0x7f014813f710 2024-03-25T18:32:58.424 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.423+0000 7f012ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:58.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.428+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628158254 0 0) 0x7f0148133710 con 0x7f014813f710 2024-03-25T18:32:58.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.428+0000 7f012ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:58.429 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.428+0000 7f012ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:58.430 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.428+0000 7f012ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:58.430 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.428+0000 7f012ffff640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:32:58.430 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.428+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0134004eb0 con 0x7f014813f710 2024-03-25T18:32:58.434 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.433+0000 7f012ffff640 10 client.?.objecter ms_handle_connect 0x7f01481430c0 2024-03-25T18:32:58.434 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.433+0000 7f012ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:58.435 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.434+0000 7f012ffff640 10 client.?.objecter ms_handle_connect 0x7f01480a98f0 2024-03-25T18:32:58.435 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.434+0000 7f012ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:58.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.449+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1143701141 0 0) 0x7f0134004eb0 con 0x7f014813f710 2024-03-25T18:32:58.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.449+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0148133710 con 0x7f014813f710 2024-03-25T18:32:58.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.449+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f013c001bc0 con 0x7f014813f710 2024-03-25T18:32:58.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.449+0000 7f012ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:58.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.449+0000 7f012ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:58.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.449+0000 7f012ffff640 10 monclient(hunting): dump: 2024-03-25T18:32:58.451 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:58.451 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:58.451 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:58.452 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:58.452 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:58.452 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:58.452 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:58.452 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:58.452 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:58.452 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:58.452 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.449+0000 7f012ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:58.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.451+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778623178 0 0) 0x7f0148139070 con 0x7f01481430c0 2024-03-25T18:32:58.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.451+0000 7f012ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:58.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.451+0000 7f012ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:58.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.451+0000 7f012ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:58.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.451+0000 7f012ffff640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:32:58.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.451+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0134004eb0 con 0x7f01481430c0 2024-03-25T18:32:58.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.452+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263871811 0 0) 0x7f01481338a0 con 0x7f01480a98f0 2024-03-25T18:32:58.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.452+0000 7f012ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:58.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.452+0000 7f012ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:58.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.452+0000 7f012ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:58.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.452+0000 7f012ffff640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:32:58.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.452+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0148139070 con 0x7f01480a98f0 2024-03-25T18:32:58.462 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3822102196 0 0) 0x7f0148133710 con 0x7f014813f710 2024-03-25T18:32:58.462 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 >> v1:172.21.15.99:6790/0 conn(0x7f01480a98f0 legacy=0x7f01481378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:58.462 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 >> v1:172.21.15.99:6789/0 conn(0x7f01481430c0 legacy=0x7f0148133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:58.462 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f012ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:58.462 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f012ffff640 1 monclient: found mon.b 2024-03-25T18:32:58.462 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f012ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:58.462 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f012ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:58.462 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0148133a30 con 0x7f014813f710 2024-03-25T18:32:58.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f012ffff640 10 monclient: _finish_auth 0 2024-03-25T18:32:58.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f012ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:58.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f012ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:58.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f014dc3e640 5 monclient: authenticate success, global_id 4567 2024-03-25T18:32:58.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f014dc3e640 10 monclient: _renew_subs 2024-03-25T18:32:58.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f014dc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:58.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f014dc3e640 1 -- 172.21.15.99:0/1034407030 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01480b7100 con 0x7f014813f710 2024-03-25T18:32:58.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f014dc3e640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:58.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f014dc3e640 10 monclient: _renew_subs 2024-03-25T18:32:58.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f014dc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:58.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f014dc3e640 1 -- 172.21.15.99:0/1034407030 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0148136e60 con 0x7f014813f710 2024-03-25T18:32:58.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.460+0000 7f014dc3e640 1 librados: init done 2024-03-25T18:32:58.463 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.461+0000 7f014dc3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:58.464 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.461+0000 7f014dc3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:58.464 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.461+0000 7f014dc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:58.464 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.461+0000 7f014dc3e640 1 -- 172.21.15.99:0/1034407030 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01480627b0 con 0x7f014813f710 2024-03-25T18:32:58.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.730+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f013c0011f0 con 0x7f014813f710 2024-03-25T18:32:58.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.730+0000 7f012ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:58.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.730+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f013c003a40 con 0x7f014813f710 2024-03-25T18:32:58.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.730+0000 7f012ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:58.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.730+0000 7f012ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:58.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.730+0000 7f012ffff640 10 monclient: dump: 2024-03-25T18:32:58.732 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:32:58.732 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:32:58.732 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:58.732 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:32:58.732 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:58.733 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:32:58.733 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:32:58.733 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:32:58.733 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:32:58.733 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:32:58.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.730+0000 7f0145ffb640 4 set_mon_vals no callback set 2024-03-25T18:32:58.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.731+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f013c003c60 con 0x7f014813f710 2024-03-25T18:32:58.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.732+0000 7f01477fe640 10 monclient: get_auth_request con 0x7f013406de10 auth_method 0 2024-03-25T18:32:58.734 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.732+0000 7f012ffff640 10 client.4567.objecter ms_handle_connect 0x7f013406de10 2024-03-25T18:32:58.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.738+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f013c054980 con 0x7f014813f710 2024-03-25T18:32:58.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.738+0000 7f012ffff640 10 client.4567.objecter ms_dispatch 0x7f0148000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:58.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.738+0000 7f012ffff640 3 client.4567.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:58.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.738+0000 7f012ffff640 3 client.4567.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:58.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.739+0000 7f012ffff640 20 client.4567.objecter dump_active .. 0 homeless 2024-03-25T18:32:58.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.739+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f013c0502c0 con 0x7f014813f710 2024-03-25T18:32:58.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.739+0000 7f012ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:58.740 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.739+0000 7f012ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:59.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.999+0000 7f014dc3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:59.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.999+0000 7f014dc3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:59.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.999+0000 7f014dc3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:32:59.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:58.999+0000 7f014dc3e640 1 -- 172.21.15.99:0/1034407030 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0148069f40 con 0x7f014813f710 2024-03-25T18:32:59.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.020+0000 7f012ffff640 1 -- 172.21.15.99:0/1034407030 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f013c0523c0 con 0x7f014813f710 2024-03-25T18:32:59.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.020+0000 7f012ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:59.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.020+0000 7f012ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:59.022 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:32:59.022 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:59.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.023+0000 7f014dc3e640 10 librados: watch_flush enter 2024-03-25T18:32:59.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.023+0000 7f014dc3e640 10 librados: watch_flush exit 2024-03-25T18:32:59.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.023+0000 7f014dc3e640 20 client.4567.objecter shutdown clearing up homeless session... 2024-03-25T18:32:59.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.023+0000 7f014dc3e640 10 client.4567.objecter successfully canceled tick 2024-03-25T18:32:59.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.023+0000 7f014dc3e640 1 -- 172.21.15.99:0/1034407030 >> v1:172.21.15.190:6800/4220409699 conn(0x7f013406de10 legacy=0x7f01340702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:59.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.023+0000 7f014dc3e640 10 monclient: shutdown 2024-03-25T18:32:59.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.023+0000 7f014dc3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:59.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.023+0000 7f014dc3e640 1 -- 172.21.15.99:0/1034407030 >> v1:172.21.15.190:6789/0 conn(0x7f014813f710 legacy=0x7f01481daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:59.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.023+0000 7f014dc3e640 1 -- 172.21.15.99:0/1034407030 shutdown_connections 2024-03-25T18:32:59.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.023+0000 7f014dc3e640 1 -- 172.21.15.99:0/1034407030 >> 172.21.15.99:0/1034407030 conn(0x7f01480b7ea0 msgr2=0x7f0148074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:59.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.023+0000 7f014dc3e640 1 -- 172.21.15.99:0/1034407030 shutdown_connections 2024-03-25T18:32:59.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.023+0000 7f014dc3e640 1 -- 172.21.15.99:0/1034407030 wait complete. 2024-03-25T18:32:59.026 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:32:59.023+0000 7f014dc3e640 1 librados: shutdown 2024-03-25T18:32:59.034 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:05.841 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:05.937 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.936+0000 7fea03086640 10 monclient: get_monmap_and_config 2024-03-25T18:33:05.937 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.936+0000 7fea03086640 10 monclient: build_initial_monmap 2024-03-25T18:33:05.937 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.936+0000 7fea03086640 10 monclient: monmap: 2024-03-25T18:33:05.937 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:05.937 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:05.937 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:33:05.937283+0000 2024-03-25T18:33:05.937 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:33:05.937283+0000 2024-03-25T18:33:05.937 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:05.938 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:05.938 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:05.938 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:33:05.938 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:33:05.938 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:05.939 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.937+0000 7fea03086640 1 Processor -- start 2024-03-25T18:33:05.939 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.937+0000 7fea03086640 1 -- start start 2024-03-25T18:33:05.939 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.937+0000 7fea03086640 10 monclient: init 2024-03-25T18:33:05.939 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.938+0000 7fea03086640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:05.939 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.938+0000 7fea03086640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:05.939 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.938+0000 7fea03086640 10 monclient(hunting): picked mon.noname-a con 0x7fe9fc061890 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:05.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.938+0000 7fea03086640 10 monclient(hunting): picked mon.noname-b con 0x7fe9fc06e740 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:05.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.938+0000 7fea03086640 10 monclient(hunting): picked mon.noname-c con 0x7fe9fc0ae280 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:05.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.938+0000 7fea03086640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc0ae830 con 0x7fe9fc061890 2024-03-25T18:33:05.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.938+0000 7fea03086640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc06f720 con 0x7fe9fc0ae280 2024-03-25T18:33:05.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.938+0000 7fea03086640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc0ada80 con 0x7fe9fc06e740 2024-03-25T18:33:05.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.938+0000 7fea03086640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:05.940 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.938+0000 7fea03086640 10 monclient(hunting): authenticate will time out at 7804.106039s 2024-03-25T18:33:05.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.954+0000 7fea00e22640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fe9fc061890 0x7fe9fc13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:49408/0 (socket says 172.21.15.99:49408) 2024-03-25T18:33:05.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.954+0000 7fea00e22640 1 -- 172.21.15.99:0/3158690585 learned_addr learned my addr 172.21.15.99:0/3158690585 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:05.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.962+0000 7fe9fb7fe640 1 -- 172.21.15.99:0/3158690585 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534807257 0 0) 0x7fe9fc0ae830 con 0x7fe9fc061890 2024-03-25T18:33:05.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.962+0000 7fe9fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:05.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.962+0000 7fe9fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:05.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.962+0000 7fe9fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:05.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.962+0000 7fe9fb7fe640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:33:05.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.962+0000 7fe9fb7fe640 1 -- 172.21.15.99:0/3158690585 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ec003520 con 0x7fe9fc061890 2024-03-25T18:33:05.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.981+0000 7fe9fb7fe640 1 -- 172.21.15.99:0/3158690585 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835538497 0 0) 0x7fe9fc06f720 con 0x7fe9fc0ae280 2024-03-25T18:33:05.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.981+0000 7fe9fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:05.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.981+0000 7fe9fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:05.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.981+0000 7fe9fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:05.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.981+0000 7fe9fb7fe640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:33:05.983 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.981+0000 7fe9fb7fe640 1 -- 172.21.15.99:0/3158690585 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9fc0ae830 con 0x7fe9fc0ae280 2024-03-25T18:33:05.988 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.986+0000 7fe9fb7fe640 1 -- 172.21.15.99:0/3158690585 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827387149 0 0) 0x7fe9fc0ada80 con 0x7fe9fc06e740 2024-03-25T18:33:05.988 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.986+0000 7fe9fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:05.988 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.986+0000 7fe9fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:05.988 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.986+0000 7fe9fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:05.988 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.986+0000 7fe9fb7fe640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:33:05.988 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.986+0000 7fe9fb7fe640 1 -- 172.21.15.99:0/3158690585 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9fc06f720 con 0x7fe9fc06e740 2024-03-25T18:33:05.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 1 -- 172.21.15.99:0/3158690585 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 786623182 0 0) 0x7fe9ec003520 con 0x7fe9fc061890 2024-03-25T18:33:05.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 1 -- 172.21.15.99:0/3158690585 >> v1:172.21.15.99:6790/0 conn(0x7fe9fc06e740 legacy=0x7fe9fc145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:05.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 1 -- 172.21.15.99:0/3158690585 >> v1:172.21.15.190:6789/0 conn(0x7fe9fc0ae280 legacy=0x7fe9fc145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:05.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:33:05.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 1 monclient: found mon.noname-a 2024-03-25T18:33:05.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:05.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:33:05.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 1 -- 172.21.15.99:0/3158690585 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9fc13fbd0 con 0x7fe9fc061890 2024-03-25T18:33:05.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:33:05.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:33:05.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:05.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 1 -- 172.21.15.99:0/3158690585 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe9f0001690 con 0x7fe9fc061890 2024-03-25T18:33:05.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:05.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:05.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fe9fb7fe640 10 monclient: dump: 2024-03-25T18:33:05.994 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:05.994 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:05.994 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:05.994 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:05.995 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:05.995 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:05.995 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:05.995 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:05.995 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:05.995 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:05.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fea03086640 5 monclient: authenticate success, global_id 4560 2024-03-25T18:33:05.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:05.992+0000 7fea03086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:06.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.011+0000 7fe9fb7fe640 1 -- 172.21.15.99:0/3158690585 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9f00017f0 con 0x7fe9fc061890 2024-03-25T18:33:06.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.011+0000 7fe9fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:06.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.011+0000 7fe9fb7fe640 1 -- 172.21.15.99:0/3158690585 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe9f0001c00 con 0x7fe9fc061890 2024-03-25T18:33:06.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.011+0000 7fea03086640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:06.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.011+0000 7fea03086640 4 set_mon_vals no callback set 2024-03-25T18:33:06.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.011+0000 7fe9fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:06.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.011+0000 7fe9fb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:06.013 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.011+0000 7fe9fb7fe640 10 monclient: dump: 2024-03-25T18:33:06.014 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:06.014 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:06.014 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:06.014 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:06.014 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:06.014 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:06.014 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:06.014 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:06.014 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:06.014 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:06.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.012+0000 7fea03086640 10 monclient: shutdown 2024-03-25T18:33:06.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.012+0000 7fea03086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.012+0000 7fea03086640 1 -- 172.21.15.99:0/3158690585 >> v1:172.21.15.99:6789/0 conn(0x7fe9fc061890 legacy=0x7fe9fc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.012+0000 7fea03086640 1 -- 172.21.15.99:0/3158690585 shutdown_connections 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.012+0000 7fea03086640 1 -- 172.21.15.99:0/3158690585 >> 172.21.15.99:0/3158690585 conn(0x7fe9fc0b0ef0 msgr2=0x7fe9fc0b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.012+0000 7fea03086640 1 -- 172.21.15.99:0/3158690585 shutdown_connections 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.012+0000 7fea03086640 1 -- 172.21.15.99:0/3158690585 wait complete. 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.013+0000 7fea03086640 10 monclient: build_initial_monmap 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.013+0000 7fea03086640 10 monclient: monmap: 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:06.017 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:33:06.018 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:33:06.018 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:06.018 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.014+0000 7fea03086640 1 librados: starting msgr at 2024-03-25T18:33:06.018 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.014+0000 7fea03086640 1 librados: starting objecter 2024-03-25T18:33:06.018 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.014+0000 7fea03086640 1 Processor -- start 2024-03-25T18:33:06.018 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.014+0000 7fea03086640 1 -- start start 2024-03-25T18:33:06.018 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.014+0000 7fea03086640 1 librados: setting wanted keys 2024-03-25T18:33:06.018 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.014+0000 7fea03086640 1 librados: calling monclient init 2024-03-25T18:33:06.018 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.014+0000 7fea03086640 10 monclient: init 2024-03-25T18:33:06.018 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.015+0000 7fea03086640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:06.018 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.015+0000 7fea03086640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:33:06.018 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.015+0000 7fea03086640 10 monclient(hunting): picked mon.noname-b con 0x7fe9fc061890 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:06.019 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.015+0000 7fea03086640 10 monclient(hunting): picked mon.noname-a con 0x7fe9fc0ae280 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:06.019 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.015+0000 7fea03086640 10 monclient(hunting): picked mon.noname-c con 0x7fe9fc142f50 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:06.019 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.015+0000 7fea03086640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc13d240 con 0x7fe9fc0ae280 2024-03-25T18:33:06.019 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.015+0000 7fea03086640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc13d790 con 0x7fe9fc061890 2024-03-25T18:33:06.019 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.015+0000 7fea03086640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc13da70 con 0x7fe9fc142f50 2024-03-25T18:33:06.019 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.015+0000 7fea03086640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:06.019 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.015+0000 7fea03086640 10 monclient(hunting): authenticate will time out at 7804.183054s 2024-03-25T18:33:06.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.021+0000 7fe9fbfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fe9fc0ae280 0x7fe9fc137160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:49442/0 (socket says 172.21.15.99:49442) 2024-03-25T18:33:06.022 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.021+0000 7fe9fbfff640 1 -- 172.21.15.99:0/4089035935 learned_addr learned my addr 172.21.15.99:0/4089035935 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:06.023 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.022+0000 7fe9f97fa640 10 client.?.objecter ms_handle_connect 0x7fe9fc0ae280 2024-03-25T18:33:06.023 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.022+0000 7fe9f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:06.024 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.022+0000 7fe9f97fa640 10 client.?.objecter ms_handle_connect 0x7fe9fc142f50 2024-03-25T18:33:06.024 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.022+0000 7fe9f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:06.024 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.023+0000 7fe9f97fa640 10 client.?.objecter ms_handle_connect 0x7fe9fc061890 2024-03-25T18:33:06.024 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.023+0000 7fe9f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:06.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.030+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24730812 0 0) 0x7fe9fc13d240 con 0x7fe9fc0ae280 2024-03-25T18:33:06.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.030+0000 7fe9f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:06.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.030+0000 7fe9f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:06.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.030+0000 7fe9f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:06.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.030+0000 7fe9f97fa640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:33:06.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.030+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e0004eb0 con 0x7fe9fc0ae280 2024-03-25T18:33:06.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.030+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884160307 0 0) 0x7fe9fc13d790 con 0x7fe9fc061890 2024-03-25T18:33:06.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.030+0000 7fe9f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:06.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.030+0000 7fe9f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:06.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.030+0000 7fe9f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:06.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.030+0000 7fe9f97fa640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:33:06.032 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.030+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9fc13d240 con 0x7fe9fc061890 2024-03-25T18:33:06.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.034+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880900736 0 0) 0x7fe9fc13da70 con 0x7fe9fc142f50 2024-03-25T18:33:06.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.034+0000 7fe9f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:06.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.034+0000 7fe9f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:06.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.034+0000 7fe9f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:06.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.034+0000 7fe9f97fa640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:33:06.036 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.034+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9fc13d790 con 0x7fe9fc142f50 2024-03-25T18:33:06.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.039+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3755262264 0 0) 0x7fe9e0004eb0 con 0x7fe9fc0ae280 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.039+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9fc13da70 con 0x7fe9fc0ae280 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.039+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe9e8001a10 con 0x7fe9fc0ae280 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.039+0000 7fe9f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.039+0000 7fe9f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.039+0000 7fe9f97fa640 10 monclient(hunting): dump: 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:06.041 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:06.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.039+0000 7fe9f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:06.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.055+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2774270689 0 0) 0x7fe9fc13d240 con 0x7fe9fc061890 2024-03-25T18:33:06.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.055+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9e0004eb0 con 0x7fe9fc061890 2024-03-25T18:33:06.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.055+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe9f0001b00 con 0x7fe9fc061890 2024-03-25T18:33:06.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.055+0000 7fe9f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:06.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.055+0000 7fe9f97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:06.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.055+0000 7fe9f97fa640 10 monclient(hunting): dump: 2024-03-25T18:33:06.056 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:06.056 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:06.056 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:06.056 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:06.056 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:06.056 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:06.057 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:06.057 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:06.057 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:06.057 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:06.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1068005103 0 0) 0x7fe9fc13da70 con 0x7fe9fc0ae280 2024-03-25T18:33:06.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 >> v1:172.21.15.99:6790/0 conn(0x7fe9fc142f50 legacy=0x7fe9fc137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:06.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 >> v1:172.21.15.190:6789/0 conn(0x7fe9fc061890 legacy=0x7fe9fc13bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:06.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fe9f97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fe9f97fa640 1 monclient: found mon.a 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fe9f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fe9f97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9fc137f80 con 0x7fe9fc0ae280 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fe9f97fa640 10 monclient: _finish_auth 0 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fe9f97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fe9f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fea03086640 5 monclient: authenticate success, global_id 4563 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fea03086640 10 monclient: _renew_subs 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fea03086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fea03086640 1 -- 172.21.15.99:0/4089035935 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9fc13d4b0 con 0x7fe9fc0ae280 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fea03086640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fea03086640 10 monclient: _renew_subs 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fea03086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fea03086640 1 -- 172.21.15.99:0/4089035935 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9fc13b410 con 0x7fe9fc0ae280 2024-03-25T18:33:06.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.062+0000 7fea03086640 1 librados: init done 2024-03-25T18:33:06.065 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.063+0000 7fea03086640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:06.065 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.063+0000 7fea03086640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:06.065 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.063+0000 7fea03086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:06.065 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.063+0000 7fea03086640 1 -- 172.21.15.99:0/4089035935 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9fc0627b0 con 0x7fe9fc0ae280 2024-03-25T18:33:06.366 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.365+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9e8001c70 con 0x7fe9fc0ae280 2024-03-25T18:33:06.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.365+0000 7fe9f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:06.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.365+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe9e8003a60 con 0x7fe9fc0ae280 2024-03-25T18:33:06.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.365+0000 7fe9f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:06.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.365+0000 7fe9f97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:06.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.365+0000 7fe9f97fa640 10 monclient: dump: 2024-03-25T18:33:06.367 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:06.367 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:06.367 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:06.367 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:06.367 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:06.367 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:06.367 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:06.367 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:06.368 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:06.368 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:06.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.365+0000 7fe9faffd640 4 set_mon_vals no callback set 2024-03-25T18:33:06.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.366+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fe9e8003c80 con 0x7fe9fc0ae280 2024-03-25T18:33:06.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.367+0000 7fea00e22640 10 monclient: get_auth_request con 0x7fe9e006df70 auth_method 0 2024-03-25T18:33:06.368 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.367+0000 7fe9f97fa640 10 client.4563.objecter ms_handle_connect 0x7fe9e006df70 2024-03-25T18:33:06.375 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.373+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fe9e8054860 con 0x7fe9fc0ae280 2024-03-25T18:33:06.375 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.373+0000 7fe9f97fa640 10 client.4563.objecter ms_dispatch 0x7fe9fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:06.375 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.373+0000 7fe9f97fa640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:06.375 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.373+0000 7fe9f97fa640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:06.375 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.374+0000 7fe9f97fa640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-25T18:33:06.375 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.374+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9e8050370 con 0x7fe9fc0ae280 2024-03-25T18:33:06.375 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.374+0000 7fe9f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:06.376 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.374+0000 7fe9f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:06.628 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.627+0000 7fea03086640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:06.628 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.627+0000 7fea03086640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:06.628 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.627+0000 7fea03086640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:06.629 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.627+0000 7fea03086640 1 -- 172.21.15.99:0/4089035935 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9fc13f050 con 0x7fe9fc0ae280 2024-03-25T18:33:06.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.656+0000 7fe9f97fa640 1 -- 172.21.15.99:0/4089035935 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fe9e80522a0 con 0x7fe9fc0ae280 2024-03-25T18:33:06.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.656+0000 7fe9f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:06.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.656+0000 7fe9f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:06.657 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:33:06.657 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:06.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.658+0000 7fea03086640 10 librados: watch_flush enter 2024-03-25T18:33:06.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.658+0000 7fea03086640 10 librados: watch_flush exit 2024-03-25T18:33:06.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.658+0000 7fea03086640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-25T18:33:06.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.658+0000 7fea03086640 10 client.4563.objecter successfully canceled tick 2024-03-25T18:33:06.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.658+0000 7fea03086640 1 -- 172.21.15.99:0/4089035935 >> v1:172.21.15.190:6800/4220409699 conn(0x7fe9e006df70 legacy=0x7fe9e0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:06.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.658+0000 7fea03086640 10 monclient: shutdown 2024-03-25T18:33:06.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.658+0000 7fea03086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:06.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.658+0000 7fea03086640 1 -- 172.21.15.99:0/4089035935 >> v1:172.21.15.99:6789/0 conn(0x7fe9fc0ae280 legacy=0x7fe9fc137160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:06.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.658+0000 7fea03086640 1 -- 172.21.15.99:0/4089035935 shutdown_connections 2024-03-25T18:33:06.661 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.658+0000 7fea03086640 1 -- 172.21.15.99:0/4089035935 >> 172.21.15.99:0/4089035935 conn(0x7fe9fc0b0ef0 msgr2=0x7fe9fc13f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:06.661 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.658+0000 7fea03086640 1 -- 172.21.15.99:0/4089035935 shutdown_connections 2024-03-25T18:33:06.661 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.658+0000 7fea03086640 1 -- 172.21.15.99:0/4089035935 wait complete. 2024-03-25T18:33:06.661 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:06.658+0000 7fea03086640 1 librados: shutdown 2024-03-25T18:33:06.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:09.797 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:08:10.389 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:09.813 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:08:10.406 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:09.840 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:08:10.433 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:09.854 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:08:10.446 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:09.869 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:08:10.428 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:09.873 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:08:10.465 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:09.879 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:08:10.437 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:09.906 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:08:10.464 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:09.913 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:08:10.506 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:13.478 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:13.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.594+0000 7f61c5515640 10 monclient: get_monmap_and_config 2024-03-25T18:33:13.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.594+0000 7f61c5515640 10 monclient: build_initial_monmap 2024-03-25T18:33:13.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.595+0000 7f61c5515640 10 monclient: monmap: 2024-03-25T18:33:13.596 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:13.596 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:13.596 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:33:13.596048+0000 2024-03-25T18:33:13.596 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:33:13.596048+0000 2024-03-25T18:33:13.597 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:13.597 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:13.597 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:13.597 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:33:13.597 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:33:13.597 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:13.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.595+0000 7f61c5515640 1 Processor -- start 2024-03-25T18:33:13.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.595+0000 7f61c5515640 1 -- start start 2024-03-25T18:33:13.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.595+0000 7f61c5515640 10 monclient: init 2024-03-25T18:33:13.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.596+0000 7f61c5515640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:13.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.596+0000 7f61c5515640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:13.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.596+0000 7f61c5515640 10 monclient(hunting): picked mon.noname-c con 0x7f61c00a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:13.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.596+0000 7f61c5515640 10 monclient(hunting): picked mon.noname-b con 0x7f61c013f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:13.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.596+0000 7f61c5515640 10 monclient(hunting): picked mon.noname-a con 0x7f61c01430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:13.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.596+0000 7f61c5515640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c0142520 con 0x7f61c01430c0 2024-03-25T18:33:13.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.596+0000 7f61c5515640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c00b28a0 con 0x7f61c00a98f0 2024-03-25T18:33:13.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.596+0000 7f61c5515640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c00b2a30 con 0x7f61c013f710 2024-03-25T18:33:13.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.596+0000 7f61c5515640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:13.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.596+0000 7f61c5515640 10 monclient(hunting): authenticate will time out at 7811.764420s 2024-03-25T18:33:13.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.599+0000 7f61bf7fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f61c01430c0 0x7f61c014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:54124/0 (socket says 172.21.15.99:54124) 2024-03-25T18:33:13.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.599+0000 7f61bf7fe640 1 -- 172.21.15.99:0/1086818304 learned_addr learned my addr 172.21.15.99:0/1086818304 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:13.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.612+0000 7f61be7fc640 1 -- 172.21.15.99:0/1086818304 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945975719 0 0) 0x7f61c0142520 con 0x7f61c01430c0 2024-03-25T18:33:13.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.612+0000 7f61be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:13.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.612+0000 7f61be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:13.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.612+0000 7f61be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:13.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.612+0000 7f61be7fc640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:33:13.613 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.612+0000 7f61be7fc640 1 -- 172.21.15.99:0/1086818304 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61ac003520 con 0x7f61c01430c0 2024-03-25T18:33:13.618 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.617+0000 7f61be7fc640 1 -- 172.21.15.99:0/1086818304 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954671212 0 0) 0x7f61c00b28a0 con 0x7f61c00a98f0 2024-03-25T18:33:13.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.617+0000 7f61be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:13.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.617+0000 7f61be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:13.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.617+0000 7f61be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:13.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.617+0000 7f61be7fc640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:33:13.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.617+0000 7f61be7fc640 1 -- 172.21.15.99:0/1086818304 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c0142520 con 0x7f61c00a98f0 2024-03-25T18:33:13.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 1 -- 172.21.15.99:0/1086818304 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1233695684 0 0) 0x7f61ac003520 con 0x7f61c01430c0 2024-03-25T18:33:13.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 1 -- 172.21.15.99:0/1086818304 >> v1:172.21.15.99:6790/0 conn(0x7f61c013f710 legacy=0x7f61c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:33:13.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 1 -- 172.21.15.99:0/1086818304 >> v1:172.21.15.190:6789/0 conn(0x7f61c00a98f0 legacy=0x7f61c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:13.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:33:13.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 1 monclient: found mon.noname-a 2024-03-25T18:33:13.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:13.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:33:13.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 1 -- 172.21.15.99:0/1086818304 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c00b44c0 con 0x7f61c01430c0 2024-03-25T18:33:13.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 10 monclient: _finish_auth 0 2024-03-25T18:33:13.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:33:13.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:13.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 1 -- 172.21.15.99:0/1086818304 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f61b8001a10 con 0x7f61c01430c0 2024-03-25T18:33:13.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:13.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:13.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61be7fc640 10 monclient: dump: 2024-03-25T18:33:13.626 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:13.626 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:13.626 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:13.626 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:13.626 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:13.626 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:13.626 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:13.626 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:13.626 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:13.626 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:13.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61c5515640 5 monclient: authenticate success, global_id 4566 2024-03-25T18:33:13.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.623+0000 7f61c5515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:13.641 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.640+0000 7f61be7fc640 1 -- 172.21.15.99:0/1086818304 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61b8001b70 con 0x7f61c01430c0 2024-03-25T18:33:13.641 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.640+0000 7f61be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:13.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.640+0000 7f61be7fc640 1 -- 172.21.15.99:0/1086818304 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f61b8001f80 con 0x7f61c01430c0 2024-03-25T18:33:13.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.640+0000 7f61be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:13.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.640+0000 7f61be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:13.642 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.640+0000 7f61be7fc640 10 monclient: dump: 2024-03-25T18:33:13.642 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:13.642 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:13.642 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:13.642 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:13.642 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:13.642 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:13.642 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:13.642 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:13.643 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:13.643 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:13.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.640+0000 7f61c5515640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:13.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.640+0000 7f61c5515640 4 set_mon_vals no callback set 2024-03-25T18:33:13.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.641+0000 7f61c5515640 10 monclient: shutdown 2024-03-25T18:33:13.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.641+0000 7f61c5515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:13.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.641+0000 7f61c5515640 1 -- 172.21.15.99:0/1086818304 >> v1:172.21.15.99:6789/0 conn(0x7f61c01430c0 legacy=0x7f61c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:13.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.641+0000 7f61c5515640 1 -- 172.21.15.99:0/1086818304 shutdown_connections 2024-03-25T18:33:13.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.641+0000 7f61c5515640 1 -- 172.21.15.99:0/1086818304 >> 172.21.15.99:0/1086818304 conn(0x7f61c00b7ea0 msgr2=0x7f61c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:13.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.641+0000 7f61c5515640 1 -- 172.21.15.99:0/1086818304 shutdown_connections 2024-03-25T18:33:13.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.641+0000 7f61c5515640 1 -- 172.21.15.99:0/1086818304 wait complete. 2024-03-25T18:33:13.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.642+0000 7f61c5515640 10 monclient: build_initial_monmap 2024-03-25T18:33:13.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.642+0000 7f61c5515640 10 monclient: monmap: 2024-03-25T18:33:13.645 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:13.645 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:13.645 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:33:13.645 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:33:13.645 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:13.645 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:33:13.645 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:13.645 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:33:13.645 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:33:13.646 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:13.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.642+0000 7f61c5515640 1 librados: starting msgr at 2024-03-25T18:33:13.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.642+0000 7f61c5515640 1 librados: starting objecter 2024-03-25T18:33:13.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.643+0000 7f61c5515640 1 Processor -- start 2024-03-25T18:33:13.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.643+0000 7f61c5515640 1 -- start start 2024-03-25T18:33:13.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.643+0000 7f61c5515640 1 librados: setting wanted keys 2024-03-25T18:33:13.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.643+0000 7f61c5515640 1 librados: calling monclient init 2024-03-25T18:33:13.646 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.643+0000 7f61c5515640 10 monclient: init 2024-03-25T18:33:13.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.644+0000 7f61c5515640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:13.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.644+0000 7f61c5515640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:13.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.644+0000 7f61c5515640 10 monclient(hunting): picked mon.noname-a con 0x7f61c00a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:13.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.644+0000 7f61c5515640 10 monclient(hunting): picked mon.noname-c con 0x7f61c013f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:13.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.644+0000 7f61c5515640 10 monclient(hunting): picked mon.noname-b con 0x7f61c01430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:13.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.644+0000 7f61c5515640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c0139780 con 0x7f61c00a98f0 2024-03-25T18:33:13.647 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.644+0000 7f61c5515640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c01386a0 con 0x7f61c01430c0 2024-03-25T18:33:13.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.644+0000 7f61c5515640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c00ab950 con 0x7f61c013f710 2024-03-25T18:33:13.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.644+0000 7f61c5515640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:13.648 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.644+0000 7f61c5515640 10 monclient(hunting): authenticate will time out at 7811.812152s 2024-03-25T18:33:13.652 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.651+0000 7f61bf7fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f61c01430c0 0x7f61c01e2bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:52378/0 (socket says 172.21.15.99:52378) 2024-03-25T18:33:13.652 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.651+0000 7f61bf7fe640 1 -- 172.21.15.99:0/2636567981 learned_addr learned my addr 172.21.15.99:0/2636567981 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:13.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.655+0000 7f61b7fff640 10 client.?.objecter ms_handle_connect 0x7f61c01430c0 2024-03-25T18:33:13.656 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.655+0000 7f61b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:13.660 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.659+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701533277 0 0) 0x7f61c01386a0 con 0x7f61c01430c0 2024-03-25T18:33:13.661 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.659+0000 7f61b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:13.661 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.659+0000 7f61b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:13.661 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.659+0000 7f61b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:13.661 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.659+0000 7f61b7fff640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:33:13.661 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.659+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a0004eb0 con 0x7f61c01430c0 2024-03-25T18:33:13.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.663+0000 7f61b7fff640 10 client.?.objecter ms_handle_connect 0x7f61c013f710 2024-03-25T18:33:13.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.663+0000 7f61b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:13.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.664+0000 7f61b7fff640 10 client.?.objecter ms_handle_connect 0x7f61c00a98f0 2024-03-25T18:33:13.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.664+0000 7f61b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:13.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.669+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 71379258 0 0) 0x7f61a0004eb0 con 0x7f61c01430c0 2024-03-25T18:33:13.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.669+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61c01386a0 con 0x7f61c01430c0 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.669+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f61b8001f40 con 0x7f61c01430c0 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.669+0000 7f61b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.669+0000 7f61b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.669+0000 7f61b7fff640 10 monclient(hunting): dump: 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:13.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.669+0000 7f61b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:13.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.674+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126497748 0 0) 0x7f61c0139780 con 0x7f61c00a98f0 2024-03-25T18:33:13.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.674+0000 7f61b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:13.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.674+0000 7f61b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:13.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.674+0000 7f61b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:13.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.674+0000 7f61b7fff640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:33:13.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.674+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a0004eb0 con 0x7f61c00a98f0 2024-03-25T18:33:13.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.676+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300128476 0 0) 0x7f61c00ab950 con 0x7f61c013f710 2024-03-25T18:33:13.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.676+0000 7f61b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:13.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.676+0000 7f61b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:13.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.676+0000 7f61b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:13.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.676+0000 7f61b7fff640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:33:13.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.676+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c0139780 con 0x7f61c013f710 2024-03-25T18:33:13.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.678+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2671386087 0 0) 0x7f61c01386a0 con 0x7f61c01430c0 2024-03-25T18:33:13.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 >> v1:172.21.15.99:6790/0 conn(0x7f61c013f710 legacy=0x7f61c01da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:13.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 >> v1:172.21.15.99:6789/0 conn(0x7f61c00a98f0 legacy=0x7f61c0133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:13.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61b7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:33:13.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61b7fff640 1 monclient: found mon.b 2024-03-25T18:33:13.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:13.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61b7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:33:13.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c00b1600 con 0x7f61c01430c0 2024-03-25T18:33:13.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61b7fff640 10 monclient: _finish_auth 0 2024-03-25T18:33:13.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61b7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:33:13.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:13.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61c5515640 5 monclient: authenticate success, global_id 4579 2024-03-25T18:33:13.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61c5515640 10 monclient: _renew_subs 2024-03-25T18:33:13.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61c5515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:33:13.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61c5515640 1 -- 172.21.15.99:0/2636567981 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c00b1100 con 0x7f61c01430c0 2024-03-25T18:33:13.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61c5515640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:13.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61c5515640 10 monclient: _renew_subs 2024-03-25T18:33:13.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61c5515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:33:13.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61c5515640 1 -- 172.21.15.99:0/2636567981 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c01378c0 con 0x7f61c01430c0 2024-03-25T18:33:13.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.679+0000 7f61c5515640 1 librados: init done 2024-03-25T18:33:13.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.680+0000 7f61c5515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:13.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.680+0000 7f61c5515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:13.683 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.680+0000 7f61c5515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:33:13.683 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.680+0000 7f61c5515640 1 -- 172.21.15.99:0/2636567981 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c01384b0 con 0x7f61c01430c0 2024-03-25T18:33:13.976 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.975+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61b8004e10 con 0x7f61c01430c0 2024-03-25T18:33:13.977 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.975+0000 7f61b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:13.977 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.975+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f61b8005120 con 0x7f61c01430c0 2024-03-25T18:33:13.977 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.975+0000 7f61b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:13.977 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.975+0000 7f61b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:13.977 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.975+0000 7f61b7fff640 10 monclient: dump: 2024-03-25T18:33:13.977 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:13.977 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:13.977 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:13.977 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:13.977 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:13.977 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:13.978 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:13.978 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:13.978 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:13.978 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:13.978 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.975+0000 7f61bdffb640 4 set_mon_vals no callback set 2024-03-25T18:33:13.979 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.976+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f61b801b0f0 con 0x7f61c01430c0 2024-03-25T18:33:13.979 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.976+0000 7f61beffd640 10 monclient: get_auth_request con 0x7f61a006de10 auth_method 0 2024-03-25T18:33:13.979 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.976+0000 7f61b7fff640 10 client.4579.objecter ms_handle_connect 0x7f61a006de10 2024-03-25T18:33:13.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.980+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f61b8054c60 con 0x7f61c01430c0 2024-03-25T18:33:13.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.980+0000 7f61b7fff640 10 client.4579.objecter ms_dispatch 0x7f61c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:13.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.980+0000 7f61b7fff640 3 client.4579.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:13.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.980+0000 7f61b7fff640 3 client.4579.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:13.982 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.980+0000 7f61b7fff640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-25T18:33:13.983 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.981+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f61b8050770 con 0x7f61c01430c0 2024-03-25T18:33:13.983 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.981+0000 7f61b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:13.983 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:13.981+0000 7f61b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:14.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.217+0000 7f61c5515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:14.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.217+0000 7f61c5515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:14.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.217+0000 7f61c5515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:33:14.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.217+0000 7f61c5515640 1 -- 172.21.15.99:0/2636567981 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61c00627b0 con 0x7f61c01430c0 2024-03-25T18:33:14.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.260+0000 7f61b7fff640 1 -- 172.21.15.99:0/2636567981 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f61b80526a0 con 0x7f61c01430c0 2024-03-25T18:33:14.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.260+0000 7f61b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:14.261 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.260+0000 7f61b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:14.262 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:33:14.262 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:14.264 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.262+0000 7f61c5515640 10 librados: watch_flush enter 2024-03-25T18:33:14.264 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.262+0000 7f61c5515640 10 librados: watch_flush exit 2024-03-25T18:33:14.264 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.262+0000 7f61c5515640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-25T18:33:14.264 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.262+0000 7f61c5515640 10 client.4579.objecter successfully canceled tick 2024-03-25T18:33:14.264 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.262+0000 7f61c5515640 1 -- 172.21.15.99:0/2636567981 >> v1:172.21.15.190:6800/4220409699 conn(0x7f61a006de10 legacy=0x7f61a00702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:14.264 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.262+0000 7f61c5515640 10 monclient: shutdown 2024-03-25T18:33:14.264 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.262+0000 7f61c5515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:14.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.262+0000 7f61c5515640 1 -- 172.21.15.99:0/2636567981 >> v1:172.21.15.190:6789/0 conn(0x7f61c01430c0 legacy=0x7f61c01e2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:14.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.262+0000 7f61c5515640 1 -- 172.21.15.99:0/2636567981 shutdown_connections 2024-03-25T18:33:14.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.262+0000 7f61c5515640 1 -- 172.21.15.99:0/2636567981 >> 172.21.15.99:0/2636567981 conn(0x7f61c00b7ea0 msgr2=0x7f61c0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:14.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.262+0000 7f61c5515640 1 -- 172.21.15.99:0/2636567981 shutdown_connections 2024-03-25T18:33:14.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.262+0000 7f61c5515640 1 -- 172.21.15.99:0/2636567981 wait complete. 2024-03-25T18:33:14.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:14.262+0000 7f61c5515640 1 librados: shutdown 2024-03-25T18:33:14.277 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:21.084 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:21.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.197+0000 7f4160a72640 10 monclient: get_monmap_and_config 2024-03-25T18:33:21.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.197+0000 7f4160a72640 10 monclient: build_initial_monmap 2024-03-25T18:33:21.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.197+0000 7f4160a72640 10 monclient: monmap: 2024-03-25T18:33:21.198 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:21.198 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:21.198 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:33:21.198483+0000 2024-03-25T18:33:21.198 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:33:21.198483+0000 2024-03-25T18:33:21.198 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:21.199 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:21.199 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:21.199 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:33:21.199 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:33:21.199 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:21.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.198+0000 7f4160a72640 1 Processor -- start 2024-03-25T18:33:21.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.198+0000 7f4160a72640 1 -- start start 2024-03-25T18:33:21.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.198+0000 7f4160a72640 10 monclient: init 2024-03-25T18:33:21.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.199+0000 7f4160a72640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:21.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.199+0000 7f4160a72640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:21.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.199+0000 7f4160a72640 10 monclient(hunting): picked mon.noname-b con 0x7f415c0a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:21.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.199+0000 7f4160a72640 10 monclient(hunting): picked mon.noname-c con 0x7f415c13f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:21.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.199+0000 7f4160a72640 10 monclient(hunting): picked mon.noname-a con 0x7f415c1430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:21.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.199+0000 7f4160a72640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c142520 con 0x7f415c1430c0 2024-03-25T18:33:21.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.199+0000 7f4160a72640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c0b28a0 con 0x7f415c13f710 2024-03-25T18:33:21.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.199+0000 7f4160a72640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c0b2a30 con 0x7f415c0a98f0 2024-03-25T18:33:21.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.199+0000 7f4160a72640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:21.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.199+0000 7f4160a72640 10 monclient(hunting): authenticate will time out at 7819.367207s 2024-03-25T18:33:21.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.203+0000 7f415a59c640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f415c0a98f0 0x7f415c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:56748/0 (socket says 172.21.15.99:56748) 2024-03-25T18:33:21.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.203+0000 7f415a59c640 1 -- 172.21.15.99:0/3684713763 learned_addr learned my addr 172.21.15.99:0/3684713763 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:21.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.221+0000 7f4159d9b640 1 -- 172.21.15.99:0/3684713763 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024866254 0 0) 0x7f415c0b28a0 con 0x7f415c13f710 2024-03-25T18:33:21.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.221+0000 7f4159d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:21.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.221+0000 7f4159d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:21.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.221+0000 7f4159d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:21.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.221+0000 7f4159d9b640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:33:21.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.221+0000 7f4159d9b640 1 -- 172.21.15.99:0/3684713763 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4148003520 con 0x7f415c13f710 2024-03-25T18:33:21.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.222+0000 7f4159d9b640 1 -- 172.21.15.99:0/3684713763 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622199310 0 0) 0x7f415c0b2a30 con 0x7f415c0a98f0 2024-03-25T18:33:21.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.222+0000 7f4159d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:21.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.222+0000 7f4159d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:21.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.222+0000 7f4159d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:21.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.222+0000 7f4159d9b640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:33:21.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.222+0000 7f4159d9b640 1 -- 172.21.15.99:0/3684713763 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c0b28a0 con 0x7f415c0a98f0 2024-03-25T18:33:21.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.227+0000 7f4159d9b640 1 -- 172.21.15.99:0/3684713763 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962460387 0 0) 0x7f415c142520 con 0x7f415c1430c0 2024-03-25T18:33:21.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.227+0000 7f4159d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:21.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.227+0000 7f4159d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:21.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.227+0000 7f4159d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:21.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.227+0000 7f4159d9b640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:33:21.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.227+0000 7f4159d9b640 1 -- 172.21.15.99:0/3684713763 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c0b2a30 con 0x7f415c1430c0 2024-03-25T18:33:21.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 1 -- 172.21.15.99:0/3684713763 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1707678877 0 0) 0x7f415c0b28a0 con 0x7f415c0a98f0 2024-03-25T18:33:21.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 1 -- 172.21.15.99:0/3684713763 >> v1:172.21.15.190:6789/0 conn(0x7f415c13f710 legacy=0x7f415c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:21.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 1 -- 172.21.15.99:0/3684713763 >> v1:172.21.15.99:6789/0 conn(0x7f415c1430c0 legacy=0x7f415c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:21.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:33:21.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 1 monclient: found mon.noname-b 2024-03-25T18:33:21.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:21.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:33:21.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 1 -- 172.21.15.99:0/3684713763 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c0b44c0 con 0x7f415c0a98f0 2024-03-25T18:33:21.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 10 monclient: _finish_auth 0 2024-03-25T18:33:21.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:33:21.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 1 -- 172.21.15.99:0/3684713763 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f414c001670 con 0x7f415c0a98f0 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.228+0000 7f4159d9b640 10 monclient: dump: 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.229+0000 7f4160a72640 5 monclient: authenticate success, global_id 4580 2024-03-25T18:33:21.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.229+0000 7f4160a72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:21.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.244+0000 7f4159d9b640 1 -- 172.21.15.99:0/3684713763 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f414c0017d0 con 0x7f415c0a98f0 2024-03-25T18:33:21.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.244+0000 7f4159d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:21.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.244+0000 7f4159d9b640 1 -- 172.21.15.99:0/3684713763 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f414c001be0 con 0x7f415c0a98f0 2024-03-25T18:33:21.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.244+0000 7f4160a72640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:21.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.244+0000 7f4160a72640 4 set_mon_vals no callback set 2024-03-25T18:33:21.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.244+0000 7f4159d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:21.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.244+0000 7f4159d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:21.246 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.244+0000 7f4159d9b640 10 monclient: dump: 2024-03-25T18:33:21.246 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:21.246 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:21.246 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:21.246 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:21.246 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:21.246 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:21.246 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:21.246 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:21.246 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:21.246 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:21.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.245+0000 7f4160a72640 10 monclient: shutdown 2024-03-25T18:33:21.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.245+0000 7f4160a72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:21.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.245+0000 7f4160a72640 1 -- 172.21.15.99:0/3684713763 >> v1:172.21.15.99:6790/0 conn(0x7f415c0a98f0 legacy=0x7f415c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:21.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.245+0000 7f4160a72640 1 -- 172.21.15.99:0/3684713763 shutdown_connections 2024-03-25T18:33:21.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.245+0000 7f4160a72640 1 -- 172.21.15.99:0/3684713763 >> 172.21.15.99:0/3684713763 conn(0x7f415c0b7ea0 msgr2=0x7f415c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:21.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.245+0000 7f4160a72640 1 -- 172.21.15.99:0/3684713763 shutdown_connections 2024-03-25T18:33:21.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.245+0000 7f4160a72640 1 -- 172.21.15.99:0/3684713763 wait complete. 2024-03-25T18:33:21.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.246+0000 7f4160a72640 10 monclient: build_initial_monmap 2024-03-25T18:33:21.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.246+0000 7f4160a72640 10 monclient: monmap: 2024-03-25T18:33:21.248 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:21.248 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:21.248 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:33:21.248 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:33:21.248 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:21.249 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:33:21.249 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:21.249 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:33:21.249 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:33:21.249 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:21.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.247+0000 7f4160a72640 1 librados: starting msgr at 2024-03-25T18:33:21.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.247+0000 7f4160a72640 1 librados: starting objecter 2024-03-25T18:33:21.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.247+0000 7f4160a72640 1 Processor -- start 2024-03-25T18:33:21.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.247+0000 7f4160a72640 1 -- start start 2024-03-25T18:33:21.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.247+0000 7f4160a72640 1 librados: setting wanted keys 2024-03-25T18:33:21.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.247+0000 7f4160a72640 1 librados: calling monclient init 2024-03-25T18:33:21.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.247+0000 7f4160a72640 10 monclient: init 2024-03-25T18:33:21.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.248+0000 7f4160a72640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:21.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.248+0000 7f4160a72640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:21.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.248+0000 7f4160a72640 10 monclient(hunting): picked mon.noname-a con 0x7f415c0a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:21.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.248+0000 7f4160a72640 10 monclient(hunting): picked mon.noname-c con 0x7f415c13f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:21.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.248+0000 7f4160a72640 10 monclient(hunting): picked mon.noname-b con 0x7f415c1430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:21.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.248+0000 7f4160a72640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c0abb90 con 0x7f415c0a98f0 2024-03-25T18:33:21.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.248+0000 7f4160a72640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c0b1840 con 0x7f415c1430c0 2024-03-25T18:33:21.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.248+0000 7f4160a72640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c0acbf0 con 0x7f415c13f710 2024-03-25T18:33:21.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.248+0000 7f4160a72640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:21.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.248+0000 7f4160a72640 10 monclient(hunting): authenticate will time out at 7819.416183s 2024-03-25T18:33:21.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.253+0000 7f415a59c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f415c0a98f0 0x7f415c138860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:54654/0 (socket says 172.21.15.99:54654) 2024-03-25T18:33:21.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.253+0000 7f4153fff640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f415c13f710 0x7f415c0ac4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:56792/0 (socket says 172.21.15.99:56792) 2024-03-25T18:33:21.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.253+0000 7f415a59c640 1 -- 172.21.15.99:0/3911152570 learned_addr learned my addr 172.21.15.99:0/3911152570 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:21.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.253+0000 7f4152ffd640 10 client.?.objecter ms_handle_connect 0x7f415c0a98f0 2024-03-25T18:33:21.254 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.253+0000 7f4152ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:21.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.253+0000 7f4152ffd640 10 client.?.objecter ms_handle_connect 0x7f415c13f710 2024-03-25T18:33:21.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.253+0000 7f4152ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:21.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.258+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900747350 0 0) 0x7f415c0acbf0 con 0x7f415c13f710 2024-03-25T18:33:21.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.258+0000 7f4152ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:21.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.258+0000 7f4152ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:21.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.258+0000 7f4152ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:21.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.258+0000 7f4152ffd640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:33:21.259 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.258+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f413c004eb0 con 0x7f415c13f710 2024-03-25T18:33:21.264 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.263+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416708144 0 0) 0x7f415c0abb90 con 0x7f415c0a98f0 2024-03-25T18:33:21.264 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.263+0000 7f4152ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:21.264 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.263+0000 7f4152ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:21.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.263+0000 7f4152ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:21.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.263+0000 7f4152ffd640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:33:21.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.263+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c0acbf0 con 0x7f415c0a98f0 2024-03-25T18:33:21.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.270+0000 7f4152ffd640 10 client.?.objecter ms_handle_connect 0x7f415c1430c0 2024-03-25T18:33:21.272 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.271+0000 7f4152ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:21.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.274+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3518587912 0 0) 0x7f415c0acbf0 con 0x7f415c0a98f0 2024-03-25T18:33:21.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.274+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f415c0abb90 con 0x7f415c0a98f0 2024-03-25T18:33:21.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.274+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f414c001be0 con 0x7f415c0a98f0 2024-03-25T18:33:21.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.274+0000 7f4152ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:21.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.274+0000 7f4152ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:21.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.274+0000 7f4152ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:21.275 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:21.276 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:21.276 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:21.276 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:21.276 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:21.276 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:21.276 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:21.276 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:21.276 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:21.276 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:21.276 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.274+0000 7f4152ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:21.280 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.279+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3189662893 0 0) 0x7f413c004eb0 con 0x7f415c13f710 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.279+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f415c0acbf0 con 0x7f415c13f710 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.279+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f41440018d0 con 0x7f415c13f710 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.279+0000 7f4152ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.279+0000 7f4152ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.279+0000 7f4152ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:21.281 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:21.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.286+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096067600 0 0) 0x7f415c0b1840 con 0x7f415c1430c0 2024-03-25T18:33:21.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.286+0000 7f4152ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:21.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.286+0000 7f4152ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:21.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.286+0000 7f4152ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:21.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.286+0000 7f4152ffd640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:33:21.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.286+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f413c004eb0 con 0x7f415c1430c0 2024-03-25T18:33:21.293 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.291+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 213736172 0 0) 0x7f415c0acbf0 con 0x7f415c13f710 2024-03-25T18:33:21.293 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.291+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 >> v1:172.21.15.190:6789/0 conn(0x7f415c1430c0 legacy=0x7f415c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:21.293 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.291+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 >> v1:172.21.15.99:6789/0 conn(0x7f415c0a98f0 legacy=0x7f415c138860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:21.293 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.291+0000 7f4152ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4152ffd640 1 monclient: found mon.c 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4152ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4152ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c0abde0 con 0x7f415c13f710 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4152ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4152ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4152ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4160a72640 5 monclient: authenticate success, global_id 4583 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4160a72640 10 monclient: _renew_subs 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4160a72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4160a72640 1 -- 172.21.15.99:0/3911152570 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415c0ac0c0 con 0x7f415c13f710 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4160a72640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4160a72640 10 monclient: _renew_subs 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4160a72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:21.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4160a72640 1 -- 172.21.15.99:0/3911152570 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f415c136580 con 0x7f415c13f710 2024-03-25T18:33:21.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.292+0000 7f4160a72640 1 librados: init done 2024-03-25T18:33:21.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.293+0000 7f4160a72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:21.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.293+0000 7f4160a72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:21.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.293+0000 7f4160a72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:21.295 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.293+0000 7f4160a72640 1 -- 172.21.15.99:0/3911152570 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415c0627b0 con 0x7f415c13f710 2024-03-25T18:33:21.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.593+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4144003610 con 0x7f415c13f710 2024-03-25T18:33:21.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.593+0000 7f4152ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:21.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.593+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4144003a60 con 0x7f415c13f710 2024-03-25T18:33:21.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.593+0000 7f4152ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:21.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.593+0000 7f4152ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:21.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.593+0000 7f4152ffd640 10 monclient: dump: 2024-03-25T18:33:21.594 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:21.594 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:21.595 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:21.595 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:21.595 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:21.595 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:21.595 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:21.595 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:21.595 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:21.595 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:21.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.593+0000 7f415959a640 4 set_mon_vals no callback set 2024-03-25T18:33:21.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.593+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f4144003c80 con 0x7f415c13f710 2024-03-25T18:33:21.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.594+0000 7f415a59c640 10 monclient: get_auth_request con 0x7f413c06df40 auth_method 0 2024-03-25T18:33:21.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.594+0000 7f4152ffd640 10 client.4583.objecter ms_handle_connect 0x7f413c06df40 2024-03-25T18:33:21.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.600+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f41440546e0 con 0x7f415c13f710 2024-03-25T18:33:21.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.600+0000 7f4152ffd640 10 client.4583.objecter ms_dispatch 0x7f415c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:21.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.600+0000 7f4152ffd640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:21.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.600+0000 7f4152ffd640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:21.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.600+0000 7f4152ffd640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-25T18:33:21.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.600+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f41440501f0 con 0x7f415c13f710 2024-03-25T18:33:21.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.600+0000 7f4152ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:21.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.600+0000 7f4152ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:21.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.850+0000 7f4160a72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:21.852 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.850+0000 7f4160a72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:21.852 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.850+0000 7f4160a72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:21.852 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.850+0000 7f4160a72640 1 -- 172.21.15.99:0/3911152570 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f415c13e150 con 0x7f415c13f710 2024-03-25T18:33:21.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.886+0000 7f4152ffd640 1 -- 172.21.15.99:0/3911152570 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f4144052120 con 0x7f415c13f710 2024-03-25T18:33:21.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.886+0000 7f4152ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:21.887 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.886+0000 7f4152ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:21.887 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:33:21.887 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:21.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.888+0000 7f4160a72640 10 librados: watch_flush enter 2024-03-25T18:33:21.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.888+0000 7f4160a72640 10 librados: watch_flush exit 2024-03-25T18:33:21.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.888+0000 7f4160a72640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-25T18:33:21.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.888+0000 7f4160a72640 10 client.4583.objecter successfully canceled tick 2024-03-25T18:33:21.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.888+0000 7f4160a72640 1 -- 172.21.15.99:0/3911152570 >> v1:172.21.15.190:6800/4220409699 conn(0x7f413c06df40 legacy=0x7f413c070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:21.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.888+0000 7f4160a72640 10 monclient: shutdown 2024-03-25T18:33:21.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.888+0000 7f4160a72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:21.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.888+0000 7f4160a72640 1 -- 172.21.15.99:0/3911152570 >> v1:172.21.15.99:6790/0 conn(0x7f415c13f710 legacy=0x7f415c0ac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:21.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.888+0000 7f4160a72640 1 -- 172.21.15.99:0/3911152570 shutdown_connections 2024-03-25T18:33:21.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.888+0000 7f4160a72640 1 -- 172.21.15.99:0/3911152570 >> 172.21.15.99:0/3911152570 conn(0x7f415c0b7ea0 msgr2=0x7f415c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:21.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.888+0000 7f4160a72640 1 -- 172.21.15.99:0/3911152570 shutdown_connections 2024-03-25T18:33:21.891 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.888+0000 7f4160a72640 1 -- 172.21.15.99:0/3911152570 wait complete. 2024-03-25T18:33:21.891 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:21.888+0000 7f4160a72640 1 librados: shutdown 2024-03-25T18:33:21.901 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:28.708 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:28.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.821+0000 7f5fdec25640 10 monclient: get_monmap_and_config 2024-03-25T18:33:28.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.821+0000 7f5fdec25640 10 monclient: build_initial_monmap 2024-03-25T18:33:28.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.821+0000 7f5fdec25640 10 monclient: monmap: 2024-03-25T18:33:28.823 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:28.823 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:28.823 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:33:28.822925+0000 2024-03-25T18:33:28.823 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:33:28.822925+0000 2024-03-25T18:33:28.823 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:28.823 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:28.823 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:28.823 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:33:28.823 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:33:28.824 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:28.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.822+0000 7f5fdec25640 1 Processor -- start 2024-03-25T18:33:28.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.822+0000 7f5fdec25640 1 -- start start 2024-03-25T18:33:28.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.822+0000 7f5fdec25640 10 monclient: init 2024-03-25T18:33:28.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.823+0000 7f5fdec25640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:28.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.823+0000 7f5fdec25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:33:28.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.823+0000 7f5fdec25640 10 monclient(hunting): picked mon.noname-a con 0x7f5fd80b4d60 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:28.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.823+0000 7f5fdec25640 10 monclient(hunting): picked mon.noname-c con 0x7f5fd80b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:28.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.823+0000 7f5fdec25640 10 monclient(hunting): picked mon.noname-b con 0x7f5fd814be90 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:28.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.823+0000 7f5fdec25640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd80af940 con 0x7f5fd80b4d60 2024-03-25T18:33:28.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.823+0000 7f5fdec25640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd80b0a70 con 0x7f5fd80b1730 2024-03-25T18:33:28.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.823+0000 7f5fdec25640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd80b0d50 con 0x7f5fd814be90 2024-03-25T18:33:28.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.823+0000 7f5fdec25640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:28.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.823+0000 7f5fdec25640 10 monclient(hunting): authenticate will time out at 7826.991712s 2024-03-25T18:33:28.827 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.824+0000 7f5fd7fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f5fd80b1730 0x7f5fd80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:34964/0 (socket says 172.21.15.99:34964) 2024-03-25T18:33:28.827 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.825+0000 7f5fd7fff640 1 -- 172.21.15.99:0/3612111747 learned_addr learned my addr 172.21.15.99:0/3612111747 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:28.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.832+0000 7f5fd77fe640 1 -- 172.21.15.99:0/3612111747 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844149422 0 0) 0x7f5fd80b0a70 con 0x7f5fd80b1730 2024-03-25T18:33:28.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.832+0000 7f5fd77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:28.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.832+0000 7f5fd77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:28.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.832+0000 7f5fd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:28.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.832+0000 7f5fd77fe640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:33:28.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.832+0000 7f5fd77fe640 1 -- 172.21.15.99:0/3612111747 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc8003520 con 0x7f5fd80b1730 2024-03-25T18:33:28.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.840+0000 7f5fd77fe640 1 -- 172.21.15.99:0/3612111747 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975841796 0 0) 0x7f5fd80af940 con 0x7f5fd80b4d60 2024-03-25T18:33:28.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.840+0000 7f5fd77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:28.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.840+0000 7f5fd77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:28.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.840+0000 7f5fd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:28.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.840+0000 7f5fd77fe640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:33:28.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.840+0000 7f5fd77fe640 1 -- 172.21.15.99:0/3612111747 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd80b0a70 con 0x7f5fd80b4d60 2024-03-25T18:33:28.843 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.842+0000 7f5fd77fe640 1 -- 172.21.15.99:0/3612111747 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903188183 0 0) 0x7f5fd80b0d50 con 0x7f5fd814be90 2024-03-25T18:33:28.843 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.842+0000 7f5fd77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:28.843 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.842+0000 7f5fd77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:28.843 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.842+0000 7f5fd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:28.843 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.842+0000 7f5fd77fe640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:33:28.844 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.842+0000 7f5fd77fe640 1 -- 172.21.15.99:0/3612111747 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd80af940 con 0x7f5fd814be90 2024-03-25T18:33:28.847 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 1 -- 172.21.15.99:0/3612111747 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1973535948 0 0) 0x7f5fc8003520 con 0x7f5fd80b1730 2024-03-25T18:33:28.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 1 -- 172.21.15.99:0/3612111747 >> v1:172.21.15.99:6790/0 conn(0x7f5fd814be90 legacy=0x7f5fd814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:28.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 1 -- 172.21.15.99:0/3612111747 >> v1:172.21.15.99:6789/0 conn(0x7f5fd80b4d60 legacy=0x7f5fd813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:28.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:33:28.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 1 monclient: found mon.noname-c 2024-03-25T18:33:28.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:28.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:33:28.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 1 -- 172.21.15.99:0/3612111747 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd80b1030 con 0x7f5fd80b1730 2024-03-25T18:33:28.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 10 monclient: _finish_auth 0 2024-03-25T18:33:28.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:33:28.848 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:28.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 1 -- 172.21.15.99:0/3612111747 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5fc4001690 con 0x7f5fd80b1730 2024-03-25T18:33:28.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:28.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:28.849 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fd77fe640 10 monclient: dump: 2024-03-25T18:33:28.849 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:28.849 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:28.849 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:28.849 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:28.849 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:28.849 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:28.849 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:28.849 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:28.849 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:28.850 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:28.850 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fdec25640 5 monclient: authenticate success, global_id 4588 2024-03-25T18:33:28.850 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.846+0000 7f5fdec25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:28.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.849+0000 7f5fd77fe640 1 -- 172.21.15.99:0/3612111747 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fc40017f0 con 0x7f5fd80b1730 2024-03-25T18:33:28.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.849+0000 7f5fd77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:28.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.849+0000 7f5fdec25640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:28.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.849+0000 7f5fdec25640 4 set_mon_vals no callback set 2024-03-25T18:33:28.852 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.849+0000 7f5fd77fe640 1 -- 172.21.15.99:0/3612111747 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5fc4001c00 con 0x7f5fd80b1730 2024-03-25T18:33:28.852 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.849+0000 7f5fd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:28.852 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.849+0000 7f5fd77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:28.852 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.849+0000 7f5fd77fe640 10 monclient: dump: 2024-03-25T18:33:28.852 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:28.852 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:28.852 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:28.852 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:28.852 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:28.852 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:28.852 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:28.852 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:28.852 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:28.853 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:28.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.850+0000 7f5fdec25640 10 monclient: shutdown 2024-03-25T18:33:28.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.850+0000 7f5fdec25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:28.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.850+0000 7f5fdec25640 1 -- 172.21.15.99:0/3612111747 >> v1:172.21.15.190:6789/0 conn(0x7f5fd80b1730 legacy=0x7f5fd80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:28.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.850+0000 7f5fdec25640 1 -- 172.21.15.99:0/3612111747 shutdown_connections 2024-03-25T18:33:28.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.850+0000 7f5fdec25640 1 -- 172.21.15.99:0/3612111747 >> 172.21.15.99:0/3612111747 conn(0x7f5fd80617a0 msgr2=0x7f5fd8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:28.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.850+0000 7f5fdec25640 1 -- 172.21.15.99:0/3612111747 shutdown_connections 2024-03-25T18:33:28.853 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.850+0000 7f5fdec25640 1 -- 172.21.15.99:0/3612111747 wait complete. 2024-03-25T18:33:28.855 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.851+0000 7f5fdec25640 10 monclient: build_initial_monmap 2024-03-25T18:33:28.855 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.851+0000 7f5fdec25640 10 monclient: monmap: 2024-03-25T18:33:28.855 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:28.855 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:28.855 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:33:28.855 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:33:28.855 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:28.855 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:33:28.855 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:28.855 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:33:28.855 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:33:28.856 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:28.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.851+0000 7f5fdec25640 1 librados: starting msgr at 2024-03-25T18:33:28.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.851+0000 7f5fdec25640 1 librados: starting objecter 2024-03-25T18:33:28.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.851+0000 7f5fdec25640 1 Processor -- start 2024-03-25T18:33:28.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.851+0000 7f5fdec25640 1 -- start start 2024-03-25T18:33:28.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.851+0000 7f5fdec25640 1 librados: setting wanted keys 2024-03-25T18:33:28.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.851+0000 7f5fdec25640 1 librados: calling monclient init 2024-03-25T18:33:28.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.851+0000 7f5fdec25640 10 monclient: init 2024-03-25T18:33:28.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.852+0000 7f5fdec25640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:28.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.852+0000 7f5fdec25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:33:28.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.852+0000 7f5fdec25640 10 monclient(hunting): picked mon.noname-a con 0x7f5fd80b1730 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:28.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.852+0000 7f5fdec25640 10 monclient(hunting): picked mon.noname-b con 0x7f5fd80b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:28.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.852+0000 7f5fdec25640 10 monclient(hunting): picked mon.noname-c con 0x7f5fd814be90 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:28.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.852+0000 7f5fdec25640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd813b360 con 0x7f5fd80b1730 2024-03-25T18:33:28.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.852+0000 7f5fdec25640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd8141e60 con 0x7f5fd80b4d60 2024-03-25T18:33:28.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.852+0000 7f5fdec25640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd8142760 con 0x7f5fd814be90 2024-03-25T18:33:28.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.852+0000 7f5fdec25640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:28.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.852+0000 7f5fdec25640 10 monclient(hunting): authenticate will time out at 7827.020276s 2024-03-25T18:33:28.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.856+0000 7f5fdc9c1640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f5fd80b1730 0x7f5fd813ac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:54102/0 (socket says 172.21.15.99:54102) 2024-03-25T18:33:28.858 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.856+0000 7f5fdc9c1640 1 -- 172.21.15.99:0/133830143 learned_addr learned my addr 172.21.15.99:0/133830143 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:28.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.861+0000 7f5fd57fa640 10 client.?.objecter ms_handle_connect 0x7f5fd80b1730 2024-03-25T18:33:28.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.861+0000 7f5fd57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:28.865 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.864+0000 7f5fd57fa640 10 client.?.objecter ms_handle_connect 0x7f5fd814be90 2024-03-25T18:33:28.865 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.864+0000 7f5fd57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:28.871 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.870+0000 7f5fd57fa640 10 client.?.objecter ms_handle_connect 0x7f5fd80b4d60 2024-03-25T18:33:28.871 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.870+0000 7f5fd57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:28.879 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.877+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257989264 0 0) 0x7f5fd813b360 con 0x7f5fd80b1730 2024-03-25T18:33:28.879 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.877+0000 7f5fd57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:28.879 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.877+0000 7f5fd57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:28.879 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.877+0000 7f5fd57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:28.879 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.877+0000 7f5fd57fa640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:33:28.879 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.877+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fbc004eb0 con 0x7f5fd80b1730 2024-03-25T18:33:28.880 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.879+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117987014 0 0) 0x7f5fd8141e60 con 0x7f5fd80b4d60 2024-03-25T18:33:28.880 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.879+0000 7f5fd57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:28.880 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.879+0000 7f5fd57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:28.880 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.879+0000 7f5fd57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:28.880 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.879+0000 7f5fd57fa640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:33:28.880 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.879+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd813b360 con 0x7f5fd80b4d60 2024-03-25T18:33:28.908 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.907+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 599610297 0 0) 0x7f5fbc004eb0 con 0x7f5fd80b1730 2024-03-25T18:33:28.909 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.907+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fd8141e60 con 0x7f5fd80b1730 2024-03-25T18:33:28.909 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.907+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5fcc001a30 con 0x7f5fd80b1730 2024-03-25T18:33:28.909 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.907+0000 7f5fd57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:28.909 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.907+0000 7f5fd57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:28.909 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.907+0000 7f5fd57fa640 10 monclient(hunting): dump: 2024-03-25T18:33:28.909 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:28.909 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:28.909 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:28.909 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:28.909 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:28.909 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:28.909 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:28.910 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:28.910 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:28.910 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:28.910 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.907+0000 7f5fd57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:28.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 262978410 0 0) 0x7f5fd8141e60 con 0x7f5fd80b1730 2024-03-25T18:33:28.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 >> v1:172.21.15.99:6790/0 conn(0x7f5fd814be90 legacy=0x7f5fd81e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:28.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 >> v1:172.21.15.190:6789/0 conn(0x7f5fd80b4d60 legacy=0x7f5fd8141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:28.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fd57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:33:28.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fd57fa640 1 monclient: found mon.a 2024-03-25T18:33:28.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fd57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:28.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fd57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:28.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd8138f60 con 0x7f5fd80b1730 2024-03-25T18:33:28.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fd57fa640 10 monclient: _finish_auth 0 2024-03-25T18:33:28.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fd57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:33:28.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fd57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:28.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fdec25640 5 monclient: authenticate success, global_id 4581 2024-03-25T18:33:28.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fdec25640 10 monclient: _renew_subs 2024-03-25T18:33:28.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fdec25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:28.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fdec25640 1 -- 172.21.15.99:0/133830143 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd8139150 con 0x7f5fd80b1730 2024-03-25T18:33:28.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fdec25640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:28.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fdec25640 10 monclient: _renew_subs 2024-03-25T18:33:28.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fdec25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:28.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fdec25640 1 -- 172.21.15.99:0/133830143 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd8143140 con 0x7f5fd80b1730 2024-03-25T18:33:28.968 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.965+0000 7f5fdec25640 1 librados: init done 2024-03-25T18:33:28.969 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.966+0000 7f5fdec25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:28.969 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.966+0000 7f5fdec25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:28.969 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.966+0000 7f5fdec25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:28.969 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:28.966+0000 7f5fdec25640 1 -- 172.21.15.99:0/133830143 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd80627b0 con 0x7f5fd80b1730 2024-03-25T18:33:29.262 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.261+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fcc001c90 con 0x7f5fd80b1730 2024-03-25T18:33:29.263 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.261+0000 7f5fd57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:29.263 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.261+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5fcc003a60 con 0x7f5fd80b1730 2024-03-25T18:33:29.263 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.261+0000 7f5fd57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:29.263 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.261+0000 7f5fd57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:29.263 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.261+0000 7f5fd57fa640 10 monclient: dump: 2024-03-25T18:33:29.263 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:29.263 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:29.263 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:29.263 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:29.263 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:29.264 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:29.264 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:29.264 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:29.264 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:29.264 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:29.264 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.261+0000 7f5fd6ffd640 4 set_mon_vals no callback set 2024-03-25T18:33:29.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.262+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f5fcc003c80 con 0x7f5fd80b1730 2024-03-25T18:33:29.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.263+0000 7f5fd7fff640 10 monclient: get_auth_request con 0x7f5fbc06dd10 auth_method 0 2024-03-25T18:33:29.265 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.263+0000 7f5fd57fa640 10 client.4581.objecter ms_handle_connect 0x7f5fbc06dd10 2024-03-25T18:33:29.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.273+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f5fcc054870 con 0x7f5fd80b1730 2024-03-25T18:33:29.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.273+0000 7f5fd57fa640 10 client.4581.objecter ms_dispatch 0x7f5fd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:29.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.273+0000 7f5fd57fa640 3 client.4581.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:29.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.273+0000 7f5fd57fa640 3 client.4581.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:29.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.274+0000 7f5fd57fa640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-25T18:33:29.276 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.274+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5fcc050380 con 0x7f5fd80b1730 2024-03-25T18:33:29.276 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.274+0000 7f5fd57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:29.276 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.274+0000 7f5fd57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:29.532 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.531+0000 7f5fdec25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:29.532 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.531+0000 7f5fdec25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:29.532 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.531+0000 7f5fdec25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:29.532 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.531+0000 7f5fdec25640 1 -- 172.21.15.99:0/133830143 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fd81395d0 con 0x7f5fd80b1730 2024-03-25T18:33:29.567 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.566+0000 7f5fd57fa640 1 -- 172.21.15.99:0/133830143 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f5fcc0522b0 con 0x7f5fd80b1730 2024-03-25T18:33:29.567 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.566+0000 7f5fd57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:29.568 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.566+0000 7f5fd57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:29.568 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:33:29.568 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:29.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.569+0000 7f5fdec25640 10 librados: watch_flush enter 2024-03-25T18:33:29.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.569+0000 7f5fdec25640 10 librados: watch_flush exit 2024-03-25T18:33:29.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.569+0000 7f5fdec25640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-25T18:33:29.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.569+0000 7f5fdec25640 10 client.4581.objecter successfully canceled tick 2024-03-25T18:33:29.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.569+0000 7f5fdec25640 1 -- 172.21.15.99:0/133830143 >> v1:172.21.15.190:6800/4220409699 conn(0x7f5fbc06dd10 legacy=0x7f5fbc0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:29.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.569+0000 7f5fdec25640 10 monclient: shutdown 2024-03-25T18:33:29.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.569+0000 7f5fdec25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:29.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.569+0000 7f5fdec25640 1 -- 172.21.15.99:0/133830143 >> v1:172.21.15.99:6789/0 conn(0x7f5fd80b1730 legacy=0x7f5fd813ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:29.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.569+0000 7f5fdec25640 1 -- 172.21.15.99:0/133830143 shutdown_connections 2024-03-25T18:33:29.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.569+0000 7f5fdec25640 1 -- 172.21.15.99:0/133830143 >> 172.21.15.99:0/133830143 conn(0x7f5fd80617a0 msgr2=0x7f5fd8137430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:29.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.569+0000 7f5fdec25640 1 -- 172.21.15.99:0/133830143 shutdown_connections 2024-03-25T18:33:29.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.569+0000 7f5fdec25640 1 -- 172.21.15.99:0/133830143 wait complete. 2024-03-25T18:33:29.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:29.569+0000 7f5fdec25640 1 librados: shutdown 2024-03-25T18:33:29.583 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:29.881 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:08:30.473 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:29.894 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:08:30.486 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:29.928 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:08:30.520 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:29.938 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:08:30.497 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:29.946 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:08:30.538 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:29.948 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:08:30.506 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:29.953 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:08:30.546 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:29.972 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:08:30.530 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:29.977 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:08:30.570 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:36.389 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:36.503 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.502+0000 7fa68c9cf640 10 monclient: get_monmap_and_config 2024-03-25T18:33:36.503 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.502+0000 7fa68c9cf640 10 monclient: build_initial_monmap 2024-03-25T18:33:36.503 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.502+0000 7fa68c9cf640 10 monclient: monmap: 2024-03-25T18:33:36.503 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:36.504 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:36.504 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:33:36.503488+0000 2024-03-25T18:33:36.504 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:33:36.503488+0000 2024-03-25T18:33:36.504 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:36.504 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:36.504 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:36.504 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:33:36.504 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:33:36.504 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:36.505 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.503+0000 7fa68c9cf640 1 Processor -- start 2024-03-25T18:33:36.505 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.503+0000 7fa68c9cf640 1 -- start start 2024-03-25T18:33:36.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.503+0000 7fa68c9cf640 10 monclient: init 2024-03-25T18:33:36.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.503+0000 7fa68c9cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:36.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.503+0000 7fa68c9cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:33:36.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.503+0000 7fa68c9cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa6880b4d60 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:36.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.503+0000 7fa68c9cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa6880b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:36.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.504+0000 7fa68c9cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa68814be90 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:36.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.504+0000 7fa68c9cf640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6880af940 con 0x7fa6880b4d60 2024-03-25T18:33:36.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.504+0000 7fa68c9cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6880b0a70 con 0x7fa6880b1730 2024-03-25T18:33:36.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.504+0000 7fa68c9cf640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6880b0d50 con 0x7fa68814be90 2024-03-25T18:33:36.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.504+0000 7fa68c9cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:36.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.504+0000 7fa68c9cf640 10 monclient(hunting): authenticate will time out at 7834.672011s 2024-03-25T18:33:36.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.508+0000 7fa68659c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa6880b4d60 0x7fa68813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40790/0 (socket says 172.21.15.99:40790) 2024-03-25T18:33:36.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.508+0000 7fa68659c640 1 -- 172.21.15.99:0/3573247916 learned_addr learned my addr 172.21.15.99:0/3573247916 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:36.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.525+0000 7fa685d9b640 1 -- 172.21.15.99:0/3573247916 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010567890 0 0) 0x7fa6880af940 con 0x7fa6880b4d60 2024-03-25T18:33:36.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.525+0000 7fa685d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:36.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.525+0000 7fa685d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:36.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.525+0000 7fa685d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:36.527 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.525+0000 7fa685d9b640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:33:36.527 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.525+0000 7fa685d9b640 1 -- 172.21.15.99:0/3573247916 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa674003520 con 0x7fa6880b4d60 2024-03-25T18:33:36.528 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.527+0000 7fa685d9b640 1 -- 172.21.15.99:0/3573247916 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164641556 0 0) 0x7fa6880b0a70 con 0x7fa6880b1730 2024-03-25T18:33:36.528 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.527+0000 7fa685d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:36.528 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.527+0000 7fa685d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:36.529 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.527+0000 7fa685d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:36.529 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.527+0000 7fa685d9b640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:33:36.529 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.527+0000 7fa685d9b640 1 -- 172.21.15.99:0/3573247916 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6880af940 con 0x7fa6880b1730 2024-03-25T18:33:36.534 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 1 -- 172.21.15.99:0/3573247916 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2593514965 0 0) 0x7fa674003520 con 0x7fa6880b4d60 2024-03-25T18:33:36.535 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 1 -- 172.21.15.99:0/3573247916 >> v1:172.21.15.99:6790/0 conn(0x7fa68814be90 legacy=0x7fa68814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:36.535 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 1 -- 172.21.15.99:0/3573247916 >> v1:172.21.15.190:6789/0 conn(0x7fa6880b1730 legacy=0x7fa6880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:36.535 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:33:36.535 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 1 monclient: found mon.noname-a 2024-03-25T18:33:36.535 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:36.535 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:33:36.535 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 1 -- 172.21.15.99:0/3573247916 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6880b1030 con 0x7fa6880b4d60 2024-03-25T18:33:36.535 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 10 monclient: _finish_auth 0 2024-03-25T18:33:36.535 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:33:36.535 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:36.535 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 1 -- 172.21.15.99:0/3573247916 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa678001690 con 0x7fa6880b4d60 2024-03-25T18:33:36.535 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:36.535 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:36.535 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa685d9b640 10 monclient: dump: 2024-03-25T18:33:36.536 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:36.536 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:36.536 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:36.536 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:36.536 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:36.536 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:36.536 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:36.536 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:36.536 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:36.536 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:36.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa68c9cf640 5 monclient: authenticate success, global_id 4584 2024-03-25T18:33:36.536 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.533+0000 7fa68c9cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:36.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.553+0000 7fa685d9b640 1 -- 172.21.15.99:0/3573247916 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6780017f0 con 0x7fa6880b4d60 2024-03-25T18:33:36.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.553+0000 7fa685d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:36.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.553+0000 7fa685d9b640 1 -- 172.21.15.99:0/3573247916 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa678001c00 con 0x7fa6880b4d60 2024-03-25T18:33:36.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.553+0000 7fa68c9cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:36.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.553+0000 7fa68c9cf640 4 set_mon_vals no callback set 2024-03-25T18:33:36.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.553+0000 7fa685d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:36.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.553+0000 7fa685d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:36.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.553+0000 7fa685d9b640 10 monclient: dump: 2024-03-25T18:33:36.555 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:36.555 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:36.555 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:36.555 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:36.555 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:36.555 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:36.555 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:36.556 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:36.556 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:36.556 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:36.558 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.554+0000 7fa68c9cf640 10 monclient: shutdown 2024-03-25T18:33:36.558 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.554+0000 7fa68c9cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:36.558 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.554+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3573247916 >> v1:172.21.15.99:6789/0 conn(0x7fa6880b4d60 legacy=0x7fa68813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:36.558 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.554+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3573247916 shutdown_connections 2024-03-25T18:33:36.558 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.554+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3573247916 >> 172.21.15.99:0/3573247916 conn(0x7fa6880617a0 msgr2=0x7fa688061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:36.558 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.554+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3573247916 shutdown_connections 2024-03-25T18:33:36.558 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.554+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3573247916 wait complete. 2024-03-25T18:33:36.558 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.555+0000 7fa68c9cf640 10 monclient: build_initial_monmap 2024-03-25T18:33:36.558 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.555+0000 7fa68c9cf640 10 monclient: monmap: 2024-03-25T18:33:36.558 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.555+0000 7fa68c9cf640 1 librados: starting msgr at 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.555+0000 7fa68c9cf640 1 librados: starting objecter 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 1 Processor -- start 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 1 -- start start 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 1 librados: setting wanted keys 2024-03-25T18:33:36.559 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 1 librados: calling monclient init 2024-03-25T18:33:36.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 10 monclient: init 2024-03-25T18:33:36.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:36.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:36.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa6880b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:36.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa6880b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:36.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa68814be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:36.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6881360b0 con 0x7fa68814be90 2024-03-25T18:33:36.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6881304e0 con 0x7fa6880b4d60 2024-03-25T18:33:36.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa688130670 con 0x7fa6880b1730 2024-03-25T18:33:36.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:36.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.556+0000 7fa68c9cf640 10 monclient(hunting): authenticate will time out at 7834.724631s 2024-03-25T18:33:36.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.565+0000 7fa68659c640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7fa6880b1730 0x7fa68812fdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:58866/0 (socket says 172.21.15.99:58866) 2024-03-25T18:33:36.566 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.565+0000 7fa68659c640 1 -- 172.21.15.99:0/3832500603 learned_addr learned my addr 172.21.15.99:0/3832500603 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:36.568 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.567+0000 7fa67effd640 10 client.?.objecter ms_handle_connect 0x7fa6880b1730 2024-03-25T18:33:36.568 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.567+0000 7fa67effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:36.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.568+0000 7fa67effd640 10 client.?.objecter ms_handle_connect 0x7fa68814be90 2024-03-25T18:33:36.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.568+0000 7fa67effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:36.575 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.574+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117161203 0 0) 0x7fa688130670 con 0x7fa6880b1730 2024-03-25T18:33:36.575 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.574+0000 7fa67effd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:36.575 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.574+0000 7fa67effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:36.575 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.574+0000 7fa67effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:36.575 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.574+0000 7fa67effd640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:33:36.575 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.574+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa668004eb0 con 0x7fa6880b1730 2024-03-25T18:33:36.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.579+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190450393 0 0) 0x7fa6881360b0 con 0x7fa68814be90 2024-03-25T18:33:36.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.579+0000 7fa67effd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:36.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.579+0000 7fa67effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:36.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.579+0000 7fa67effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:36.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.579+0000 7fa67effd640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:33:36.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.579+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa688130670 con 0x7fa68814be90 2024-03-25T18:33:36.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.579+0000 7fa67effd640 10 client.?.objecter ms_handle_connect 0x7fa6880b4d60 2024-03-25T18:33:36.581 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.579+0000 7fa67effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.590+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 383547712 0 0) 0x7fa668004eb0 con 0x7fa6880b1730 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.590+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6881360b0 con 0x7fa6880b1730 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.590+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa678001c40 con 0x7fa6880b1730 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.590+0000 7fa67effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.590+0000 7fa67effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.590+0000 7fa67effd640 10 monclient(hunting): dump: 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:36.592 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:36.593 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:36.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.590+0000 7fa67effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:36.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.597+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2065241089 0 0) 0x7fa688130670 con 0x7fa68814be90 2024-03-25T18:33:36.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.597+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa668004eb0 con 0x7fa68814be90 2024-03-25T18:33:36.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.597+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa680001b00 con 0x7fa68814be90 2024-03-25T18:33:36.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.598+0000 7fa67effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:36.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.598+0000 7fa67effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:36.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.598+0000 7fa67effd640 10 monclient(hunting): dump: 2024-03-25T18:33:36.599 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:36.599 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:36.599 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:36.599 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:36.599 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:36.599 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:36.599 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:36.600 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:36.600 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:36.600 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:36.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.599+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176213361 0 0) 0x7fa6881304e0 con 0x7fa6880b4d60 2024-03-25T18:33:36.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.599+0000 7fa67effd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:36.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.599+0000 7fa67effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:36.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.599+0000 7fa67effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:36.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.599+0000 7fa67effd640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:33:36.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.599+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa688130670 con 0x7fa6880b4d60 2024-03-25T18:33:36.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3924240497 0 0) 0x7fa6881360b0 con 0x7fa6880b1730 2024-03-25T18:33:36.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 >> v1:172.21.15.190:6789/0 conn(0x7fa6880b4d60 legacy=0x7fa6881d79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:36.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 >> v1:172.21.15.99:6789/0 conn(0x7fa68814be90 legacy=0x7fa6881d80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:36.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa67effd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:36.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa67effd640 1 monclient: found mon.c 2024-03-25T18:33:36.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa67effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:36.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa67effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:36.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa688130800 con 0x7fa6880b1730 2024-03-25T18:33:36.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa67effd640 10 monclient: _finish_auth 0 2024-03-25T18:33:36.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa67effd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:36.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa67effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:36.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa68c9cf640 5 monclient: authenticate success, global_id 4592 2024-03-25T18:33:36.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa68c9cf640 10 monclient: _renew_subs 2024-03-25T18:33:36.606 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa68c9cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:36.606 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3832500603 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa688135b00 con 0x7fa6880b1730 2024-03-25T18:33:36.606 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa68c9cf640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:36.606 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa68c9cf640 10 monclient: _renew_subs 2024-03-25T18:33:36.606 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa68c9cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:36.606 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3832500603 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6881322c0 con 0x7fa6880b1730 2024-03-25T18:33:36.606 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.603+0000 7fa68c9cf640 1 librados: init done 2024-03-25T18:33:36.606 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.604+0000 7fa68c9cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:36.607 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.604+0000 7fa68c9cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:36.607 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.604+0000 7fa68c9cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:36.607 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.604+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3832500603 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6880b2270 con 0x7fa6880b1730 2024-03-25T18:33:36.903 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.902+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa678004a90 con 0x7fa6880b1730 2024-03-25T18:33:36.904 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.902+0000 7fa67effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:36.904 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.902+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa678004de0 con 0x7fa6880b1730 2024-03-25T18:33:36.904 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.902+0000 7fa67effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:36.904 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.902+0000 7fa67effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:36.904 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.902+0000 7fa67effd640 10 monclient: dump: 2024-03-25T18:33:36.904 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:36.904 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:36.904 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:36.904 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:36.904 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:36.904 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:36.904 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:36.905 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:36.905 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:36.905 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:36.905 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.902+0000 7fa68559a640 4 set_mon_vals no callback set 2024-03-25T18:33:36.905 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.903+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fa678005000 con 0x7fa6880b1730 2024-03-25T18:33:36.905 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.904+0000 7fa67ffff640 10 monclient: get_auth_request con 0x7fa66806df50 auth_method 0 2024-03-25T18:33:36.905 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.904+0000 7fa67effd640 10 client.4592.objecter ms_handle_connect 0x7fa66806df50 2024-03-25T18:33:36.912 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.911+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fa6780549b0 con 0x7fa6880b1730 2024-03-25T18:33:36.912 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.911+0000 7fa67effd640 10 client.4592.objecter ms_dispatch 0x7fa688000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:36.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.911+0000 7fa67effd640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:36.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.911+0000 7fa67effd640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:36.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.911+0000 7fa67effd640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-25T18:33:36.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.911+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6780504c0 con 0x7fa6880b1730 2024-03-25T18:33:36.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.911+0000 7fa67effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:36.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:36.911+0000 7fa67effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:37.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.165+0000 7fa68c9cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:37.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.165+0000 7fa68c9cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:37.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.165+0000 7fa68c9cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:37.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.165+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3832500603 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa688135ea0 con 0x7fa6880b1730 2024-03-25T18:33:37.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.188+0000 7fa67effd640 1 -- 172.21.15.99:0/3832500603 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fa6780523f0 con 0x7fa6880b1730 2024-03-25T18:33:37.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.188+0000 7fa67effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:37.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.188+0000 7fa67effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:37.190 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:33:37.190 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:37.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.191+0000 7fa68c9cf640 10 librados: watch_flush enter 2024-03-25T18:33:37.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.191+0000 7fa68c9cf640 10 librados: watch_flush exit 2024-03-25T18:33:37.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.191+0000 7fa68c9cf640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-25T18:33:37.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.191+0000 7fa68c9cf640 10 client.4592.objecter successfully canceled tick 2024-03-25T18:33:37.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.191+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3832500603 >> v1:172.21.15.190:6800/4220409699 conn(0x7fa66806df50 legacy=0x7fa668070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:37.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.191+0000 7fa68c9cf640 10 monclient: shutdown 2024-03-25T18:33:37.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.191+0000 7fa68c9cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:37.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.191+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3832500603 >> v1:172.21.15.99:6790/0 conn(0x7fa6880b1730 legacy=0x7fa68812fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:37.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.191+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3832500603 shutdown_connections 2024-03-25T18:33:37.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.191+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3832500603 >> 172.21.15.99:0/3832500603 conn(0x7fa6880617a0 msgr2=0x7fa688138500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:37.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.191+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3832500603 shutdown_connections 2024-03-25T18:33:37.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.191+0000 7fa68c9cf640 1 -- 172.21.15.99:0/3832500603 wait complete. 2024-03-25T18:33:37.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:37.191+0000 7fa68c9cf640 1 librados: shutdown 2024-03-25T18:33:37.204 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:44.011 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:44.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.127+0000 7f4b9ed2b640 10 monclient: get_monmap_and_config 2024-03-25T18:33:44.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.127+0000 7f4b9ed2b640 10 monclient: build_initial_monmap 2024-03-25T18:33:44.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.127+0000 7f4b9ed2b640 10 monclient: monmap: 2024-03-25T18:33:44.128 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:44.128 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:44.129 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:33:44.128172+0000 2024-03-25T18:33:44.129 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:33:44.128172+0000 2024-03-25T18:33:44.129 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:44.129 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:44.129 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:44.129 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:33:44.129 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:33:44.129 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:44.130 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.127+0000 7f4b9ed2b640 1 Processor -- start 2024-03-25T18:33:44.131 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.128+0000 7f4b9ed2b640 1 -- start start 2024-03-25T18:33:44.131 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.128+0000 7f4b9ed2b640 10 monclient: init 2024-03-25T18:33:44.131 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.128+0000 7f4b9ed2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:44.131 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.128+0000 7f4b9ed2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:44.131 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.128+0000 7f4b9ed2b640 10 monclient(hunting): picked mon.noname-c con 0x7f4b980a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:44.131 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.128+0000 7f4b9ed2b640 10 monclient(hunting): picked mon.noname-b con 0x7f4b9813f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:44.131 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.128+0000 7f4b9ed2b640 10 monclient(hunting): picked mon.noname-a con 0x7f4b981430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:44.131 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.128+0000 7f4b9ed2b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b98142520 con 0x7f4b981430c0 2024-03-25T18:33:44.131 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.129+0000 7f4b9ed2b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b980b28a0 con 0x7f4b980a98f0 2024-03-25T18:33:44.131 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.129+0000 7f4b9ed2b640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b980b2a30 con 0x7f4b9813f710 2024-03-25T18:33:44.132 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.129+0000 7f4b9ed2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:44.132 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.129+0000 7f4b9ed2b640 10 monclient(hunting): authenticate will time out at 7842.296896s 2024-03-25T18:33:44.132 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.131+0000 7f4b9cac7640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f4b980a98f0 0x7f4b980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:60130/0 (socket says 172.21.15.99:60130) 2024-03-25T18:33:44.133 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.131+0000 7f4b9cac7640 1 -- 172.21.15.99:0/1420033264 learned_addr learned my addr 172.21.15.99:0/1420033264 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:44.141 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.140+0000 7f4b977fe640 1 -- 172.21.15.99:0/1420033264 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627380133 0 0) 0x7f4b980b28a0 con 0x7f4b980a98f0 2024-03-25T18:33:44.142 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.140+0000 7f4b977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:44.142 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.140+0000 7f4b977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:44.142 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.140+0000 7f4b977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:44.142 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.140+0000 7f4b977fe640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:33:44.142 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.140+0000 7f4b977fe640 1 -- 172.21.15.99:0/1420033264 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b88003520 con 0x7f4b980a98f0 2024-03-25T18:33:44.147 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.146+0000 7f4b977fe640 1 -- 172.21.15.99:0/1420033264 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262672058 0 0) 0x7f4b98142520 con 0x7f4b981430c0 2024-03-25T18:33:44.147 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.146+0000 7f4b977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:33:44.147 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.146+0000 7f4b977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:44.147 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.146+0000 7f4b977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:44.147 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.146+0000 7f4b977fe640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:33:44.148 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.146+0000 7f4b977fe640 1 -- 172.21.15.99:0/1420033264 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b980b28a0 con 0x7f4b981430c0 2024-03-25T18:33:44.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 1 -- 172.21.15.99:0/1420033264 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3626000119 0 0) 0x7f4b980b28a0 con 0x7f4b981430c0 2024-03-25T18:33:44.162 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 1 -- 172.21.15.99:0/1420033264 >> v1:172.21.15.99:6790/0 conn(0x7f4b9813f710 legacy=0x7f4b981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:44.162 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 1 -- 172.21.15.99:0/1420033264 >> v1:172.21.15.190:6789/0 conn(0x7f4b980a98f0 legacy=0x7f4b980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:44.162 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 10 monclient: _finish_hunting 0 2024-03-25T18:33:44.162 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 1 monclient: found mon.noname-a 2024-03-25T18:33:44.162 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:44.162 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:33:44.162 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 1 -- 172.21.15.99:0/1420033264 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b980b44c0 con 0x7f4b981430c0 2024-03-25T18:33:44.162 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 10 monclient: _finish_auth 0 2024-03-25T18:33:44.162 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 10 monclient: _check_auth_tickets 2024-03-25T18:33:44.163 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:44.163 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 1 -- 172.21.15.99:0/1420033264 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4b900019f0 con 0x7f4b981430c0 2024-03-25T18:33:44.163 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:44.163 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:44.163 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b977fe640 10 monclient: dump: 2024-03-25T18:33:44.163 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:44.163 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:44.163 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:44.163 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:44.163 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:44.163 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:44.164 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:44.164 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:44.164 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:44.164 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:44.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b9ed2b640 5 monclient: authenticate success, global_id 4590 2024-03-25T18:33:44.164 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.160+0000 7f4b9ed2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:44.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.171+0000 7f4b977fe640 1 -- 172.21.15.99:0/1420033264 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b90001b50 con 0x7f4b981430c0 2024-03-25T18:33:44.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.171+0000 7f4b977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:44.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.171+0000 7f4b977fe640 1 -- 172.21.15.99:0/1420033264 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4b90001f20 con 0x7f4b981430c0 2024-03-25T18:33:44.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.171+0000 7f4b9ed2b640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:44.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.171+0000 7f4b9ed2b640 4 set_mon_vals no callback set 2024-03-25T18:33:44.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.171+0000 7f4b977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:44.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.171+0000 7f4b977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:44.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.171+0000 7f4b977fe640 10 monclient: dump: 2024-03-25T18:33:44.174 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:44.174 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:44.174 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:44.174 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:44.174 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:44.174 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:44.174 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:44.174 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:44.174 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:44.175 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:44.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.172+0000 7f4b9ed2b640 10 monclient: shutdown 2024-03-25T18:33:44.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.172+0000 7f4b9ed2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:44.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.172+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/1420033264 >> v1:172.21.15.99:6789/0 conn(0x7f4b981430c0 legacy=0x7f4b9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:44.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.172+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/1420033264 shutdown_connections 2024-03-25T18:33:44.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.172+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/1420033264 >> 172.21.15.99:0/1420033264 conn(0x7f4b980b7ea0 msgr2=0x7f4b980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:44.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.172+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/1420033264 shutdown_connections 2024-03-25T18:33:44.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.172+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/1420033264 wait complete. 2024-03-25T18:33:44.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.173+0000 7f4b9ed2b640 10 monclient: build_initial_monmap 2024-03-25T18:33:44.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.173+0000 7f4b9ed2b640 10 monclient: monmap: 2024-03-25T18:33:44.179 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:44.179 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:44.179 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:33:44.179 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:33:44.179 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:44.179 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:33:44.179 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:44.179 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:33:44.179 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:33:44.179 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:44.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.173+0000 7f4b9ed2b640 1 librados: starting msgr at 2024-03-25T18:33:44.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.173+0000 7f4b9ed2b640 1 librados: starting objecter 2024-03-25T18:33:44.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.174+0000 7f4b9ed2b640 1 Processor -- start 2024-03-25T18:33:44.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.174+0000 7f4b9ed2b640 1 -- start start 2024-03-25T18:33:44.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.174+0000 7f4b9ed2b640 1 librados: setting wanted keys 2024-03-25T18:33:44.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.174+0000 7f4b9ed2b640 1 librados: calling monclient init 2024-03-25T18:33:44.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.174+0000 7f4b9ed2b640 10 monclient: init 2024-03-25T18:33:44.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.174+0000 7f4b9ed2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:44.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.174+0000 7f4b9ed2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:33:44.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.174+0000 7f4b9ed2b640 10 monclient(hunting): picked mon.noname-b con 0x7f4b980a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:44.181 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.175+0000 7f4b9ed2b640 10 monclient(hunting): picked mon.noname-a con 0x7f4b9813f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:44.181 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.175+0000 7f4b9ed2b640 10 monclient(hunting): picked mon.noname-c con 0x7f4b981430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:44.181 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.175+0000 7f4b9ed2b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b98138070 con 0x7f4b9813f710 2024-03-25T18:33:44.181 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.175+0000 7f4b9ed2b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b98139800 con 0x7f4b980a98f0 2024-03-25T18:33:44.181 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.175+0000 7f4b9ed2b640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b98138980 con 0x7f4b981430c0 2024-03-25T18:33:44.181 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.175+0000 7f4b9ed2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:44.181 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.175+0000 7f4b9ed2b640 10 monclient(hunting): authenticate will time out at 7842.343224s 2024-03-25T18:33:44.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.181+0000 7f4b97fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f4b9813f710 0x7f4b981d7620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45702/0 (socket says 172.21.15.99:45702) 2024-03-25T18:33:44.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.181+0000 7f4b97fff640 1 -- 172.21.15.99:0/2993399301 learned_addr learned my addr 172.21.15.99:0/2993399301 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:44.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.183+0000 7f4b957fa640 10 client.?.objecter ms_handle_connect 0x7f4b9813f710 2024-03-25T18:33:44.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.183+0000 7f4b957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:44.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.187+0000 7f4b957fa640 10 client.?.objecter ms_handle_connect 0x7f4b980a98f0 2024-03-25T18:33:44.189 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.187+0000 7f4b957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:44.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.196+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350797115 0 0) 0x7f4b98139800 con 0x7f4b980a98f0 2024-03-25T18:33:44.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.196+0000 7f4b957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:44.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.196+0000 7f4b957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:44.198 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.196+0000 7f4b957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:44.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.196+0000 7f4b957fa640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:33:44.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.197+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b7c004eb0 con 0x7f4b980a98f0 2024-03-25T18:33:44.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.197+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150108439 0 0) 0x7f4b98138070 con 0x7f4b9813f710 2024-03-25T18:33:44.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.197+0000 7f4b957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:44.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.197+0000 7f4b957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:44.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.197+0000 7f4b957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:44.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.197+0000 7f4b957fa640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:33:44.199 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.197+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b98139800 con 0x7f4b9813f710 2024-03-25T18:33:44.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.200+0000 7f4b957fa640 10 client.?.objecter ms_handle_connect 0x7f4b981430c0 2024-03-25T18:33:44.202 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.200+0000 7f4b957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:44.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.206+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521556483 0 0) 0x7f4b98138980 con 0x7f4b981430c0 2024-03-25T18:33:44.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.206+0000 7f4b957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:33:44.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.206+0000 7f4b957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:44.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.206+0000 7f4b957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:44.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.206+0000 7f4b957fa640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:33:44.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.206+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b98138070 con 0x7f4b981430c0 2024-03-25T18:33:44.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.211+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 981193305 0 0) 0x7f4b98139800 con 0x7f4b9813f710 2024-03-25T18:33:44.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.211+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b98138980 con 0x7f4b9813f710 2024-03-25T18:33:44.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.211+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4b840019f0 con 0x7f4b9813f710 2024-03-25T18:33:44.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.211+0000 7f4b957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:44.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.211+0000 7f4b957fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:44.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.211+0000 7f4b957fa640 10 monclient(hunting): dump: 2024-03-25T18:33:44.213 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:44.213 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:44.213 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:44.214 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:44.214 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:44.214 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:44.214 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:44.214 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:44.214 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:44.214 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:44.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.211+0000 7f4b957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:44.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.218+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 326315052 0 0) 0x7f4b98138070 con 0x7f4b981430c0 2024-03-25T18:33:44.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.218+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b98139800 con 0x7f4b981430c0 2024-03-25T18:33:44.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.218+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4b90001e60 con 0x7f4b981430c0 2024-03-25T18:33:44.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.218+0000 7f4b957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:44.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.218+0000 7f4b957fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:44.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.218+0000 7f4b957fa640 10 monclient(hunting): dump: 2024-03-25T18:33:44.219 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:44.220 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:44.220 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:44.220 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:44.220 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:44.220 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:44.220 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:44.220 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:44.220 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:44.220 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:44.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 40066199 0 0) 0x7f4b98138980 con 0x7f4b9813f710 2024-03-25T18:33:44.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 >> v1:172.21.15.99:6790/0 conn(0x7f4b981430c0 legacy=0x7f4b981d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:44.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 >> v1:172.21.15.190:6789/0 conn(0x7f4b980a98f0 legacy=0x7f4b98137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:44.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b957fa640 10 monclient: _finish_hunting 0 2024-03-25T18:33:44.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b957fa640 1 monclient: found mon.a 2024-03-25T18:33:44.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:44.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b957fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:44.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b98138220 con 0x7f4b9813f710 2024-03-25T18:33:44.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b957fa640 10 monclient: _finish_auth 0 2024-03-25T18:33:44.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b957fa640 10 monclient: _check_auth_tickets 2024-03-25T18:33:44.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:44.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b9ed2b640 5 monclient: authenticate success, global_id 4593 2024-03-25T18:33:44.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b9ed2b640 10 monclient: _renew_subs 2024-03-25T18:33:44.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b9ed2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:44.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/2993399301 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b981384d0 con 0x7f4b9813f710 2024-03-25T18:33:44.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b9ed2b640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:44.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b9ed2b640 10 monclient: _renew_subs 2024-03-25T18:33:44.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b9ed2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:44.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/2993399301 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b981e31c0 con 0x7f4b9813f710 2024-03-25T18:33:44.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.219+0000 7f4b9ed2b640 1 librados: init done 2024-03-25T18:33:44.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.220+0000 7f4b9ed2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:44.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.220+0000 7f4b9ed2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:44.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.220+0000 7f4b9ed2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:44.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.220+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/2993399301 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b980627b0 con 0x7f4b9813f710 2024-03-25T18:33:44.518 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.517+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b84003610 con 0x7f4b9813f710 2024-03-25T18:33:44.519 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.517+0000 7f4b957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:44.519 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.517+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f4b84003960 con 0x7f4b9813f710 2024-03-25T18:33:44.519 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.517+0000 7f4b957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:44.519 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.517+0000 7f4b957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:44.519 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.517+0000 7f4b957fa640 10 monclient: dump: 2024-03-25T18:33:44.519 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:44.519 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:44.519 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:44.519 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:44.520 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:44.520 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:44.520 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:44.520 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:44.520 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:44.520 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:44.520 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.517+0000 7f4b96ffd640 4 set_mon_vals no callback set 2024-03-25T18:33:44.521 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.518+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f4b84003b80 con 0x7f4b9813f710 2024-03-25T18:33:44.521 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.518+0000 7f4b9cac7640 10 monclient: get_auth_request con 0x7f4b7c06df80 auth_method 0 2024-03-25T18:33:44.521 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.519+0000 7f4b957fa640 10 client.4593.objecter ms_handle_connect 0x7f4b7c06df80 2024-03-25T18:33:44.527 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.525+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f4b84054860 con 0x7f4b9813f710 2024-03-25T18:33:44.527 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.525+0000 7f4b957fa640 10 client.4593.objecter ms_dispatch 0x7f4b98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:44.527 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.525+0000 7f4b957fa640 3 client.4593.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:44.527 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.525+0000 7f4b957fa640 3 client.4593.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:44.527 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.525+0000 7f4b957fa640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-25T18:33:44.527 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.525+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b84050370 con 0x7f4b9813f710 2024-03-25T18:33:44.528 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.525+0000 7f4b957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:44.528 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.525+0000 7f4b957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:44.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.771+0000 7f4b9ed2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:44.773 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.771+0000 7f4b9ed2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:44.773 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.771+0000 7f4b9ed2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:33:44.773 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.771+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/2993399301 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b9813b430 con 0x7f4b9813f710 2024-03-25T18:33:44.805 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.803+0000 7f4b957fa640 1 -- 172.21.15.99:0/2993399301 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f4b840522a0 con 0x7f4b9813f710 2024-03-25T18:33:44.805 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.803+0000 7f4b957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:44.805 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.803+0000 7f4b957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:44.805 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:33:44.805 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:44.807 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.806+0000 7f4b9ed2b640 10 librados: watch_flush enter 2024-03-25T18:33:44.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.806+0000 7f4b9ed2b640 10 librados: watch_flush exit 2024-03-25T18:33:44.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.806+0000 7f4b9ed2b640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-25T18:33:44.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.806+0000 7f4b9ed2b640 10 client.4593.objecter successfully canceled tick 2024-03-25T18:33:44.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.806+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/2993399301 >> v1:172.21.15.190:6800/4220409699 conn(0x7f4b7c06df80 legacy=0x7f4b7c070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:44.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.806+0000 7f4b9ed2b640 10 monclient: shutdown 2024-03-25T18:33:44.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.806+0000 7f4b9ed2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:44.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.806+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/2993399301 >> v1:172.21.15.99:6789/0 conn(0x7f4b9813f710 legacy=0x7f4b981d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:44.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.806+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/2993399301 shutdown_connections 2024-03-25T18:33:44.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.806+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/2993399301 >> 172.21.15.99:0/2993399301 conn(0x7f4b980b7ea0 msgr2=0x7f4b98074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:44.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.806+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/2993399301 shutdown_connections 2024-03-25T18:33:44.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.806+0000 7f4b9ed2b640 1 -- 172.21.15.99:0/2993399301 wait complete. 2024-03-25T18:33:44.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:44.806+0000 7f4b9ed2b640 1 librados: shutdown 2024-03-25T18:33:44.820 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:49.974 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:08:50.567 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:49.986 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:08:50.578 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:50.004 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:08:50.597 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:50.037 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:08:50.630 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:50.038 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:08:50.597 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:50.045 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:08:50.638 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:50.051 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:08:50.609 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:50.078 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:08:50.670 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:50.081 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:08:50.640 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:51.625 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:51.741 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.740+0000 7fa01a5e2640 10 monclient: get_monmap_and_config 2024-03-25T18:33:51.742 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.740+0000 7fa01a5e2640 10 monclient: build_initial_monmap 2024-03-25T18:33:51.742 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.740+0000 7fa01a5e2640 10 monclient: monmap: 2024-03-25T18:33:51.742 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:51.742 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:51.742 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:33:51.741794+0000 2024-03-25T18:33:51.742 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:33:51.741794+0000 2024-03-25T18:33:51.742 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:51.742 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:51.742 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:51.742 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:33:51.742 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:33:51.742 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:51.743 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.741+0000 7fa01a5e2640 1 Processor -- start 2024-03-25T18:33:51.743 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.741+0000 7fa01a5e2640 1 -- start start 2024-03-25T18:33:51.743 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.741+0000 7fa01a5e2640 10 monclient: init 2024-03-25T18:33:51.743 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.742+0000 7fa01a5e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:51.744 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.742+0000 7fa01a5e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:51.744 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.742+0000 7fa01a5e2640 10 monclient(hunting): picked mon.noname-c con 0x7fa0140a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:51.744 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.742+0000 7fa01a5e2640 10 monclient(hunting): picked mon.noname-b con 0x7fa01413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:51.744 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.742+0000 7fa01a5e2640 10 monclient(hunting): picked mon.noname-a con 0x7fa0141430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:51.744 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.742+0000 7fa01a5e2640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa014142520 con 0x7fa0141430c0 2024-03-25T18:33:51.744 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.742+0000 7fa01a5e2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0140b28a0 con 0x7fa0140a98f0 2024-03-25T18:33:51.744 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.742+0000 7fa01a5e2640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0140b2a30 con 0x7fa01413f710 2024-03-25T18:33:51.744 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.742+0000 7fa01a5e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:51.744 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.742+0000 7fa01a5e2640 10 monclient(hunting): authenticate will time out at 7849.910138s 2024-03-25T18:33:51.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.753+0000 7fa018b7f640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa0141430c0 0x7fa01414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57876/0 (socket says 172.21.15.99:57876) 2024-03-25T18:33:51.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.753+0000 7fa018b7f640 1 -- 172.21.15.99:0/1987546602 learned_addr learned my addr 172.21.15.99:0/1987546602 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:51.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.763+0000 7fa012ffd640 1 -- 172.21.15.99:0/1987546602 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49817091 0 0) 0x7fa0140b28a0 con 0x7fa0140a98f0 2024-03-25T18:33:51.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.763+0000 7fa012ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:51.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.763+0000 7fa012ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:51.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.763+0000 7fa012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:51.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.763+0000 7fa012ffd640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:33:51.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.763+0000 7fa012ffd640 1 -- 172.21.15.99:0/1987546602 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ffc003520 con 0x7fa0140a98f0 2024-03-25T18:33:51.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.766+0000 7fa012ffd640 1 -- 172.21.15.99:0/1987546602 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858217138 0 0) 0x7fa0140b2a30 con 0x7fa01413f710 2024-03-25T18:33:51.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.766+0000 7fa012ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:51.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.766+0000 7fa012ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:51.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.766+0000 7fa012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:51.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.766+0000 7fa012ffd640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:33:51.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.766+0000 7fa012ffd640 1 -- 172.21.15.99:0/1987546602 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0140b28a0 con 0x7fa01413f710 2024-03-25T18:33:51.771 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.770+0000 7fa012ffd640 1 -- 172.21.15.99:0/1987546602 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472591475 0 0) 0x7fa014142520 con 0x7fa0141430c0 2024-03-25T18:33:51.771 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.770+0000 7fa012ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:51.771 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.770+0000 7fa012ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:51.771 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.770+0000 7fa012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:51.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.770+0000 7fa012ffd640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:33:51.772 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.770+0000 7fa012ffd640 1 -- 172.21.15.99:0/1987546602 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0140b2a30 con 0x7fa0141430c0 2024-03-25T18:33:51.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 1 -- 172.21.15.99:0/1987546602 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 881755806 0 0) 0x7f9ffc003520 con 0x7fa0140a98f0 2024-03-25T18:33:51.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 1 -- 172.21.15.99:0/1987546602 >> v1:172.21.15.99:6790/0 conn(0x7fa01413f710 legacy=0x7fa0141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 1 -- 172.21.15.99:0/1987546602 >> v1:172.21.15.99:6789/0 conn(0x7fa0141430c0 legacy=0x7fa01414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 1 monclient: found mon.noname-c 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 1 -- 172.21.15.99:0/1987546602 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0140b44c0 con 0x7fa0140a98f0 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa01a5e2640 5 monclient: authenticate success, global_id 4606 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 1 -- 172.21.15.99:0/1987546602 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa004001690 con 0x7fa0140a98f0 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa012ffd640 10 monclient: dump: 2024-03-25T18:33:51.779 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:51.780 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:51.780 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:51.780 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:51.780 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:51.780 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:51.780 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:51.780 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:51.780 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:51.780 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:51.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.777+0000 7fa01a5e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:51.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.782+0000 7fa012ffd640 1 -- 172.21.15.99:0/1987546602 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0040017f0 con 0x7fa0140a98f0 2024-03-25T18:33:51.783 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.782+0000 7fa012ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.782+0000 7fa012ffd640 1 -- 172.21.15.99:0/1987546602 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa004001c00 con 0x7fa0140a98f0 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.782+0000 7fa012ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.783+0000 7fa012ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.783+0000 7fa012ffd640 10 monclient: dump: 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:51.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.783+0000 7fa01a5e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:51.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.783+0000 7fa01a5e2640 4 set_mon_vals no callback set 2024-03-25T18:33:51.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.783+0000 7fa01a5e2640 10 monclient: shutdown 2024-03-25T18:33:51.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.783+0000 7fa01a5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:51.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.783+0000 7fa01a5e2640 1 -- 172.21.15.99:0/1987546602 >> v1:172.21.15.190:6789/0 conn(0x7fa0140a98f0 legacy=0x7fa0140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:51.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.783+0000 7fa01a5e2640 1 -- 172.21.15.99:0/1987546602 shutdown_connections 2024-03-25T18:33:51.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.783+0000 7fa01a5e2640 1 -- 172.21.15.99:0/1987546602 >> 172.21.15.99:0/1987546602 conn(0x7fa0140b7ea0 msgr2=0x7fa0140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:51.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.784+0000 7fa01a5e2640 1 -- 172.21.15.99:0/1987546602 shutdown_connections 2024-03-25T18:33:51.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.784+0000 7fa01a5e2640 1 -- 172.21.15.99:0/1987546602 wait complete. 2024-03-25T18:33:51.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.784+0000 7fa01a5e2640 10 monclient: build_initial_monmap 2024-03-25T18:33:51.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.784+0000 7fa01a5e2640 10 monclient: monmap: 2024-03-25T18:33:51.786 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:51.786 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:51.786 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:33:51.786 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:33:51.786 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:51.786 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:33:51.786 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:51.787 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:33:51.787 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:33:51.787 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:51.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.785+0000 7fa01a5e2640 1 librados: starting msgr at 2024-03-25T18:33:51.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.785+0000 7fa01a5e2640 1 librados: starting objecter 2024-03-25T18:33:51.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.785+0000 7fa01a5e2640 1 Processor -- start 2024-03-25T18:33:51.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.785+0000 7fa01a5e2640 1 -- start start 2024-03-25T18:33:51.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.785+0000 7fa01a5e2640 1 librados: setting wanted keys 2024-03-25T18:33:51.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.785+0000 7fa01a5e2640 1 librados: calling monclient init 2024-03-25T18:33:51.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.785+0000 7fa01a5e2640 10 monclient: init 2024-03-25T18:33:51.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.786+0000 7fa01a5e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:51.789 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.786+0000 7fa01a5e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:51.789 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.786+0000 7fa01a5e2640 10 monclient(hunting): picked mon.noname-c con 0x7fa01413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:51.789 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.786+0000 7fa01a5e2640 10 monclient(hunting): picked mon.noname-a con 0x7fa0141430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:51.789 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.786+0000 7fa01a5e2640 10 monclient(hunting): picked mon.noname-b con 0x7fa014139450 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:51.789 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.786+0000 7fa01a5e2640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa014138f30 con 0x7fa0141430c0 2024-03-25T18:33:51.789 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.786+0000 7fa01a5e2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa014136b60 con 0x7fa014139450 2024-03-25T18:33:51.789 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.786+0000 7fa01a5e2640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa014136cf0 con 0x7fa01413f710 2024-03-25T18:33:51.789 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.786+0000 7fa01a5e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:51.789 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.786+0000 7fa01a5e2640 10 monclient(hunting): authenticate will time out at 7849.954393s 2024-03-25T18:33:51.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.802+0000 7fa0137fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa0141430c0 0x7fa0140ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57918/0 (socket says 172.21.15.99:57918) 2024-03-25T18:33:51.803 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.802+0000 7fa0137fe640 1 -- 172.21.15.99:0/676537893 learned_addr learned my addr 172.21.15.99:0/676537893 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:51.806 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.805+0000 7fa010ff9640 10 client.?.objecter ms_handle_connect 0x7fa014139450 2024-03-25T18:33:51.806 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.805+0000 7fa010ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:51.806 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.805+0000 7fa010ff9640 10 client.?.objecter ms_handle_connect 0x7fa0141430c0 2024-03-25T18:33:51.806 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.805+0000 7fa010ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:51.815 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.814+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431139105 0 0) 0x7fa014136b60 con 0x7fa014139450 2024-03-25T18:33:51.815 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.814+0000 7fa010ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:51.815 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.814+0000 7fa010ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:51.815 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.814+0000 7fa010ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:51.815 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.814+0000 7fa010ff9640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:33:51.815 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.814+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa000004eb0 con 0x7fa014139450 2024-03-25T18:33:51.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.816+0000 7fa010ff9640 10 client.?.objecter ms_handle_connect 0x7fa01413f710 2024-03-25T18:33:51.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.816+0000 7fa010ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:51.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.818+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325859935 0 0) 0x7fa014138f30 con 0x7fa0141430c0 2024-03-25T18:33:51.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.818+0000 7fa010ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:51.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.818+0000 7fa010ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:51.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.818+0000 7fa010ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:51.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.818+0000 7fa010ff9640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:33:51.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.818+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa014136b60 con 0x7fa0141430c0 2024-03-25T18:33:51.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.823+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054279091 0 0) 0x7fa014136cf0 con 0x7fa01413f710 2024-03-25T18:33:51.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.823+0000 7fa010ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:51.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.823+0000 7fa010ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:51.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.823+0000 7fa010ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:51.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.823+0000 7fa010ff9640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:33:51.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.823+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa014138f30 con 0x7fa01413f710 2024-03-25T18:33:51.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.834+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4218507230 0 0) 0x7fa000004eb0 con 0x7fa014139450 2024-03-25T18:33:51.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.834+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa014136cf0 con 0x7fa014139450 2024-03-25T18:33:51.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.834+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa00c001bd0 con 0x7fa014139450 2024-03-25T18:33:51.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.834+0000 7fa010ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:51.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.834+0000 7fa010ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:51.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.834+0000 7fa010ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:51.835 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:51.835 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:51.835 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:51.835 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:51.835 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:51.835 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:51.836 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:51.836 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:51.836 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:51.836 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:51.836 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.834+0000 7fa010ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:51.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.838+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1829132443 0 0) 0x7fa014136b60 con 0x7fa0141430c0 2024-03-25T18:33:51.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.838+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa000004eb0 con 0x7fa0141430c0 2024-03-25T18:33:51.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.838+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa0080019b0 con 0x7fa0141430c0 2024-03-25T18:33:51.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.838+0000 7fa010ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:51.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.838+0000 7fa010ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:51.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.838+0000 7fa010ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:51.839 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:51.839 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:51.839 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:51.840 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:51.840 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:51.840 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:51.840 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:51.840 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:51.840 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:51.840 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:51.843 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.842+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2966623017 0 0) 0x7fa014138f30 con 0x7fa01413f710 2024-03-25T18:33:51.843 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.842+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa014136b60 con 0x7fa01413f710 2024-03-25T18:33:51.843 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.842+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa004001b00 con 0x7fa01413f710 2024-03-25T18:33:51.843 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.842+0000 7fa010ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:51.843 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.842+0000 7fa010ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:51.843 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.842+0000 7fa010ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:51.843 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:51.843 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:51.843 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:51.843 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:51.844 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:51.844 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:51.844 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:51.844 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:51.844 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:51.844 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:51.855 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3203814448 0 0) 0x7fa014136b60 con 0x7fa01413f710 2024-03-25T18:33:51.855 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 >> v1:172.21.15.190:6789/0 conn(0x7fa014139450 legacy=0x7fa0141e2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:51.855 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 >> v1:172.21.15.99:6789/0 conn(0x7fa0141430c0 legacy=0x7fa0140ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa010ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa010ff9640 1 monclient: found mon.c 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa010ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa010ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa014138a80 con 0x7fa01413f710 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa010ff9640 10 monclient: _finish_auth 0 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa010ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa010ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa01a5e2640 5 monclient: authenticate success, global_id 4604 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa01a5e2640 10 monclient: _renew_subs 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa01a5e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa01a5e2640 1 -- 172.21.15.99:0/676537893 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa014138cd0 con 0x7fa01413f710 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa01a5e2640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa01a5e2640 10 monclient: _renew_subs 2024-03-25T18:33:51.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa01a5e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:51.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa01a5e2640 1 -- 172.21.15.99:0/676537893 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0140ad000 con 0x7fa01413f710 2024-03-25T18:33:51.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.854+0000 7fa01a5e2640 1 librados: init done 2024-03-25T18:33:51.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.855+0000 7fa01a5e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:51.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.855+0000 7fa01a5e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:51.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.855+0000 7fa01a5e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:51.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:51.855+0000 7fa01a5e2640 1 -- 172.21.15.99:0/676537893 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0140b3ef0 con 0x7fa01413f710 2024-03-25T18:33:52.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.164+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa004001c60 con 0x7fa01413f710 2024-03-25T18:33:52.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.164+0000 7fa010ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:52.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.164+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fa004003a40 con 0x7fa01413f710 2024-03-25T18:33:52.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.164+0000 7fa010ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:52.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.164+0000 7fa010ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:52.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.164+0000 7fa010ff9640 10 monclient: dump: 2024-03-25T18:33:52.166 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:52.166 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:52.166 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:52.166 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:52.166 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:52.166 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:52.166 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:52.166 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:52.166 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:52.167 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:52.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.164+0000 7fa0127fc640 4 set_mon_vals no callback set 2024-03-25T18:33:52.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.165+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fa004019a10 con 0x7fa01413f710 2024-03-25T18:33:52.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.166+0000 7fa0137fe640 10 monclient: get_auth_request con 0x7fa00006e100 auth_method 0 2024-03-25T18:33:52.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.166+0000 7fa010ff9640 10 client.4604.objecter ms_handle_connect 0x7fa00006e100 2024-03-25T18:33:52.171 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.170+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fa004053510 con 0x7fa01413f710 2024-03-25T18:33:52.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.170+0000 7fa010ff9640 10 client.4604.objecter ms_dispatch 0x7fa014000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:52.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.170+0000 7fa010ff9640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:52.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.170+0000 7fa010ff9640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:52.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.170+0000 7fa010ff9640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-25T18:33:52.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.170+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa00404f020 con 0x7fa01413f710 2024-03-25T18:33:52.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.170+0000 7fa010ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:52.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.170+0000 7fa010ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:52.410 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.409+0000 7fa01a5e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:52.410 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.409+0000 7fa01a5e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:52.410 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.409+0000 7fa01a5e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:33:52.410 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.409+0000 7fa01a5e2640 1 -- 172.21.15.99:0/676537893 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa014139170 con 0x7fa01413f710 2024-03-25T18:33:52.466 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.465+0000 7fa010ff9640 1 -- 172.21.15.99:0/676537893 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fa014139170 con 0x7fa01413f710 2024-03-25T18:33:52.466 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.465+0000 7fa010ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:52.466 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.465+0000 7fa010ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:52.466 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:33:52.466 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:52.468 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.467+0000 7fa01a5e2640 10 librados: watch_flush enter 2024-03-25T18:33:52.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.467+0000 7fa01a5e2640 10 librados: watch_flush exit 2024-03-25T18:33:52.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.467+0000 7fa01a5e2640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-25T18:33:52.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.467+0000 7fa01a5e2640 10 client.4604.objecter successfully canceled tick 2024-03-25T18:33:52.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.467+0000 7fa01a5e2640 1 -- 172.21.15.99:0/676537893 >> v1:172.21.15.190:6800/4220409699 conn(0x7fa00006e100 legacy=0x7fa0000705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:52.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.467+0000 7fa01a5e2640 10 monclient: shutdown 2024-03-25T18:33:52.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.467+0000 7fa01a5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:52.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.467+0000 7fa01a5e2640 1 -- 172.21.15.99:0/676537893 >> v1:172.21.15.99:6790/0 conn(0x7fa01413f710 legacy=0x7fa014138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:52.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.467+0000 7fa01a5e2640 1 -- 172.21.15.99:0/676537893 shutdown_connections 2024-03-25T18:33:52.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.467+0000 7fa01a5e2640 1 -- 172.21.15.99:0/676537893 >> 172.21.15.99:0/676537893 conn(0x7fa0140b7ea0 msgr2=0x7fa0140b5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:52.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.467+0000 7fa01a5e2640 1 -- 172.21.15.99:0/676537893 shutdown_connections 2024-03-25T18:33:52.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.467+0000 7fa01a5e2640 1 -- 172.21.15.99:0/676537893 wait complete. 2024-03-25T18:33:52.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:52.467+0000 7fa01a5e2640 1 librados: shutdown 2024-03-25T18:33:52.481 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:59.287 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:59.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.400+0000 7fbccc1cf640 10 monclient: get_monmap_and_config 2024-03-25T18:33:59.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.400+0000 7fbccc1cf640 10 monclient: build_initial_monmap 2024-03-25T18:33:59.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.400+0000 7fbccc1cf640 10 monclient: monmap: 2024-03-25T18:33:59.401 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:59.401 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:59.401 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:33:59.401142+0000 2024-03-25T18:33:59.401 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:33:59.401142+0000 2024-03-25T18:33:59.401 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:59.401 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:59.401 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:59.402 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:33:59.402 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:33:59.402 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:59.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.400+0000 7fbccc1cf640 1 Processor -- start 2024-03-25T18:33:59.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.401+0000 7fbccc1cf640 1 -- start start 2024-03-25T18:33:59.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.401+0000 7fbccc1cf640 10 monclient: init 2024-03-25T18:33:59.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.401+0000 7fbccc1cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:59.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.401+0000 7fbccc1cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:59.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.401+0000 7fbccc1cf640 10 monclient(hunting): picked mon.noname-c con 0x7fbcc40705b0 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:59.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.401+0000 7fbccc1cf640 10 monclient(hunting): picked mon.noname-b con 0x7fbcc41402d0 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:59.403 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.401+0000 7fbccc1cf640 10 monclient(hunting): picked mon.noname-a con 0x7fbcc4149ae0 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:59.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.401+0000 7fbccc1cf640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc40718c0 con 0x7fbcc4149ae0 2024-03-25T18:33:59.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.401+0000 7fbccc1cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc40714c0 con 0x7fbcc40705b0 2024-03-25T18:33:59.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.401+0000 7fbccc1cf640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc4072b50 con 0x7fbcc41402d0 2024-03-25T18:33:59.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.401+0000 7fbccc1cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:59.404 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.401+0000 7fbccc1cf640 10 monclient(hunting): authenticate will time out at 7857.569692s 2024-03-25T18:33:59.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.406+0000 7fbcc9f6b640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fbcc40705b0 0x7fbcc413d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:55254/0 (socket says 172.21.15.99:55254) 2024-03-25T18:33:59.407 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.406+0000 7fbcc9f6b640 1 -- 172.21.15.99:0/1619720234 learned_addr learned my addr 172.21.15.99:0/1619720234 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:59.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.425+0000 7fbcc8f69640 1 -- 172.21.15.99:0/1619720234 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527658480 0 0) 0x7fbcc40714c0 con 0x7fbcc40705b0 2024-03-25T18:33:59.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.425+0000 7fbcc8f69640 10 monclient(hunting): _negotiate 2024-03-25T18:33:59.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.425+0000 7fbcc8f69640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:59.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.425+0000 7fbcc8f69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:59.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.425+0000 7fbcc8f69640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:33:59.426 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.425+0000 7fbcc8f69640 1 -- 172.21.15.99:0/1619720234 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcb0003520 con 0x7fbcc40705b0 2024-03-25T18:33:59.427 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.426+0000 7fbcc8f69640 1 -- 172.21.15.99:0/1619720234 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233826144 0 0) 0x7fbcc40718c0 con 0x7fbcc4149ae0 2024-03-25T18:33:59.427 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.426+0000 7fbcc8f69640 10 monclient(hunting): _negotiate 2024-03-25T18:33:59.427 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.426+0000 7fbcc8f69640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:59.427 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.426+0000 7fbcc8f69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:59.427 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.426+0000 7fbcc8f69640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:33:59.427 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.426+0000 7fbcc8f69640 1 -- 172.21.15.99:0/1619720234 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcc40714c0 con 0x7fbcc4149ae0 2024-03-25T18:33:59.439 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 1 -- 172.21.15.99:0/1619720234 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925945430 0 0) 0x7fbcc4072b50 con 0x7fbcc41402d0 2024-03-25T18:33:59.439 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 10 monclient(hunting): _negotiate 2024-03-25T18:33:59.439 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:59.439 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:59.439 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:33:59.439 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 1 -- 172.21.15.99:0/1619720234 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcc40718c0 con 0x7fbcc41402d0 2024-03-25T18:33:59.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 1 -- 172.21.15.99:0/1619720234 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 937071952 0 0) 0x7fbcb0003520 con 0x7fbcc40705b0 2024-03-25T18:33:59.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 1 -- 172.21.15.99:0/1619720234 >> v1:172.21.15.99:6790/0 conn(0x7fbcc41402d0 legacy=0x7fbcc4140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:59.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 1 -- 172.21.15.99:0/1619720234 >> v1:172.21.15.99:6789/0 conn(0x7fbcc4149ae0 legacy=0x7fbcc414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:59.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 10 monclient: _finish_hunting 0 2024-03-25T18:33:59.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 1 monclient: found mon.noname-c 2024-03-25T18:33:59.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:59.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:33:59.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 1 -- 172.21.15.99:0/1619720234 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc413dd10 con 0x7fbcc40705b0 2024-03-25T18:33:59.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 10 monclient: _finish_auth 0 2024-03-25T18:33:59.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 10 monclient: _check_auth_tickets 2024-03-25T18:33:59.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:59.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 1 -- 172.21.15.99:0/1619720234 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbcc0001610 con 0x7fbcc40705b0 2024-03-25T18:33:59.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:59.440 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:59.441 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbcc8f69640 10 monclient: dump: 2024-03-25T18:33:59.441 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:59.441 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:59.441 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:59.441 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:59.441 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:59.441 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:59.441 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:59.441 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:59.441 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:59.441 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:59.441 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbccc1cf640 5 monclient: authenticate success, global_id 4612 2024-03-25T18:33:59.441 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.438+0000 7fbccc1cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:59.446 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.445+0000 7fbcc8f69640 1 -- 172.21.15.99:0/1619720234 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcc0001770 con 0x7fbcc40705b0 2024-03-25T18:33:59.446 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.445+0000 7fbcc8f69640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:59.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.445+0000 7fbcc8f69640 1 -- 172.21.15.99:0/1619720234 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbcc0001bc0 con 0x7fbcc40705b0 2024-03-25T18:33:59.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.445+0000 7fbccc1cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:59.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.445+0000 7fbccc1cf640 4 set_mon_vals no callback set 2024-03-25T18:33:59.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.445+0000 7fbcc8f69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:59.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.445+0000 7fbcc8f69640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:59.447 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.445+0000 7fbcc8f69640 10 monclient: dump: 2024-03-25T18:33:59.447 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:59.447 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:59.447 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:59.447 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:59.447 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:59.447 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:59.447 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:59.447 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:59.448 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:59.448 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:59.449 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.446+0000 7fbccc1cf640 10 monclient: shutdown 2024-03-25T18:33:59.449 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.446+0000 7fbccc1cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:59.449 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.446+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1619720234 >> v1:172.21.15.190:6789/0 conn(0x7fbcc40705b0 legacy=0x7fbcc413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:59.449 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.446+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1619720234 shutdown_connections 2024-03-25T18:33:59.449 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.446+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1619720234 >> 172.21.15.99:0/1619720234 conn(0x7fbcc40b5d10 msgr2=0x7fbcc40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:59.449 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.446+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1619720234 shutdown_connections 2024-03-25T18:33:59.449 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.446+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1619720234 wait complete. 2024-03-25T18:33:59.449 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.447+0000 7fbccc1cf640 10 monclient: build_initial_monmap 2024-03-25T18:33:59.450 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.447+0000 7fbccc1cf640 10 monclient: monmap: 2024-03-25T18:33:59.450 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:33:59.450 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:59.450 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:33:59.450 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:33:59.450 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:59.450 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:33:59.450 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:33:59.450 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:33:59.450 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:33:59.450 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:59.450 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.448+0000 7fbccc1cf640 1 librados: starting msgr at 2024-03-25T18:33:59.450 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.448+0000 7fbccc1cf640 1 librados: starting objecter 2024-03-25T18:33:59.450 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.448+0000 7fbccc1cf640 1 Processor -- start 2024-03-25T18:33:59.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.448+0000 7fbccc1cf640 1 -- start start 2024-03-25T18:33:59.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.448+0000 7fbccc1cf640 1 librados: setting wanted keys 2024-03-25T18:33:59.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.448+0000 7fbccc1cf640 1 librados: calling monclient init 2024-03-25T18:33:59.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.448+0000 7fbccc1cf640 10 monclient: init 2024-03-25T18:33:59.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.449+0000 7fbccc1cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:59.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.449+0000 7fbccc1cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:59.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.449+0000 7fbccc1cf640 10 monclient(hunting): picked mon.noname-c con 0x7fbcc40705b0 addr v1:172.21.15.99:6790/0 2024-03-25T18:33:59.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.449+0000 7fbccc1cf640 10 monclient(hunting): picked mon.noname-a con 0x7fbcc41402d0 addr v1:172.21.15.99:6789/0 2024-03-25T18:33:59.451 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.449+0000 7fbccc1cf640 10 monclient(hunting): picked mon.noname-b con 0x7fbcc4149ae0 addr v1:172.21.15.190:6789/0 2024-03-25T18:33:59.452 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.449+0000 7fbccc1cf640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc413dfd0 con 0x7fbcc41402d0 2024-03-25T18:33:59.452 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.449+0000 7fbccc1cf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc4143f50 con 0x7fbcc4149ae0 2024-03-25T18:33:59.452 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.449+0000 7fbccc1cf640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc4144230 con 0x7fbcc40705b0 2024-03-25T18:33:59.452 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.449+0000 7fbccc1cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:59.452 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.449+0000 7fbccc1cf640 10 monclient(hunting): authenticate will time out at 7857.617297s 2024-03-25T18:33:59.460 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.459+0000 7fbcca76c640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fbcc4149ae0 0x7fbcc4133da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:55266/0 (socket says 172.21.15.99:55266) 2024-03-25T18:33:59.460 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.459+0000 7fbcca76c640 1 -- 172.21.15.99:0/1826371649 learned_addr learned my addr 172.21.15.99:0/1826371649 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:33:59.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.460+0000 7fbcba7fc640 10 client.?.objecter ms_handle_connect 0x7fbcc4149ae0 2024-03-25T18:33:59.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.460+0000 7fbcba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:59.466 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.465+0000 7fbcba7fc640 10 client.?.objecter ms_handle_connect 0x7fbcc40705b0 2024-03-25T18:33:59.466 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.465+0000 7fbcba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:59.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.470+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750043506 0 0) 0x7fbcc4143f50 con 0x7fbcc4149ae0 2024-03-25T18:33:59.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.470+0000 7fbcba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:59.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.470+0000 7fbcba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:59.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.470+0000 7fbcba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:59.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.470+0000 7fbcba7fc640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:33:59.471 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.470+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca8004eb0 con 0x7fbcc4149ae0 2024-03-25T18:33:59.475 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.474+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038872345 0 0) 0x7fbcc4144230 con 0x7fbcc40705b0 2024-03-25T18:33:59.475 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.474+0000 7fbcba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:59.475 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.474+0000 7fbcba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:59.476 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.474+0000 7fbcba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:59.476 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.474+0000 7fbcba7fc640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:33:59.476 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.474+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcc4143f50 con 0x7fbcc40705b0 2024-03-25T18:33:59.477 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.476+0000 7fbcba7fc640 10 client.?.objecter ms_handle_connect 0x7fbcc41402d0 2024-03-25T18:33:59.477 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.476+0000 7fbcba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:59.480 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.479+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1832807043 0 0) 0x7fbca8004eb0 con 0x7fbcc4149ae0 2024-03-25T18:33:59.480 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.479+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbcc4144230 con 0x7fbcc4149ae0 2024-03-25T18:33:59.481 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.479+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbcbc001cf0 con 0x7fbcc4149ae0 2024-03-25T18:33:59.481 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.479+0000 7fbcba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:59.481 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.479+0000 7fbcba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:59.481 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.479+0000 7fbcba7fc640 10 monclient(hunting): dump: 2024-03-25T18:33:59.481 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:59.481 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:59.481 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:59.481 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:59.481 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:59.481 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:59.481 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:59.481 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:59.482 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:59.482 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:59.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.479+0000 7fbcba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:59.488 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.486+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 275780790 0 0) 0x7fbcc4143f50 con 0x7fbcc40705b0 2024-03-25T18:33:59.488 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.486+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbca8004eb0 con 0x7fbcc40705b0 2024-03-25T18:33:59.488 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.487+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbcc0001b40 con 0x7fbcc40705b0 2024-03-25T18:33:59.488 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.487+0000 7fbcba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:59.488 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.487+0000 7fbcba7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:59.488 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.487+0000 7fbcba7fc640 10 monclient(hunting): dump: 2024-03-25T18:33:59.488 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:59.488 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:59.488 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:59.488 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:59.489 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:59.489 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:59.489 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:59.489 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:59.489 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:59.489 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:59.489 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.488+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692526053 0 0) 0x7fbcc413dfd0 con 0x7fbcc41402d0 2024-03-25T18:33:59.489 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.488+0000 7fbcba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:59.490 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.488+0000 7fbcba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:59.490 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.488+0000 7fbcba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:59.490 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.488+0000 7fbcba7fc640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:33:59.490 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.488+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcc4143f50 con 0x7fbcc41402d0 2024-03-25T18:33:59.498 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2657907438 0 0) 0x7fbcc4144230 con 0x7fbcc4149ae0 2024-03-25T18:33:59.498 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 >> v1:172.21.15.99:6790/0 conn(0x7fbcc40705b0 legacy=0x7fbcc41424d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:59.498 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 >> v1:172.21.15.99:6789/0 conn(0x7fbcc41402d0 legacy=0x7fbcc41d8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:59.498 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbcba7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:33:59.499 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbcba7fc640 1 monclient: found mon.b 2024-03-25T18:33:59.499 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbcba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:59.499 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbcba7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:33:59.499 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc4142be0 con 0x7fbcc4149ae0 2024-03-25T18:33:59.499 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbcba7fc640 10 monclient: _finish_auth 0 2024-03-25T18:33:59.499 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbcba7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:33:59.499 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbcba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:59.499 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbccc1cf640 5 monclient: authenticate success, global_id 4615 2024-03-25T18:33:59.499 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbccc1cf640 10 monclient: _renew_subs 2024-03-25T18:33:59.499 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbccc1cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:33:59.499 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1826371649 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc4142e90 con 0x7fbcc4149ae0 2024-03-25T18:33:59.500 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbccc1cf640 10 client.4615.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:59.500 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbccc1cf640 10 monclient: _renew_subs 2024-03-25T18:33:59.500 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbccc1cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:33:59.500 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1826371649 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc41d96c0 con 0x7fbcc4149ae0 2024-03-25T18:33:59.500 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.497+0000 7fbccc1cf640 1 librados: init done 2024-03-25T18:33:59.500 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.498+0000 7fbccc1cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:59.500 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.498+0000 7fbccc1cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:59.500 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.498+0000 7fbccc1cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:33:59.500 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.498+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1826371649 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc40627b0 con 0x7fbcc4149ae0 2024-03-25T18:33:59.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.783+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcbc003970 con 0x7fbcc4149ae0 2024-03-25T18:33:59.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.783+0000 7fbcba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:59.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.783+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fbcbc003dc0 con 0x7fbcc4149ae0 2024-03-25T18:33:59.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.783+0000 7fbcba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:59.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.783+0000 7fbcba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:59.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.783+0000 7fbcba7fc640 10 monclient: dump: 2024-03-25T18:33:59.785 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:33:59.785 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:33:59.785 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:59.785 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:33:59.785 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:59.785 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:33:59.785 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:33:59.785 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:33:59.785 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:33:59.785 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:33:59.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.783+0000 7fbcbbfff640 4 set_mon_vals no callback set 2024-03-25T18:33:59.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.783+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fbcbc003fe0 con 0x7fbcc4149ae0 2024-03-25T18:33:59.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.784+0000 7fbcc9f6b640 10 monclient: get_auth_request con 0x7fbca806df00 auth_method 0 2024-03-25T18:33:59.786 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.784+0000 7fbcba7fc640 10 client.4615.objecter ms_handle_connect 0x7fbca806df00 2024-03-25T18:33:59.790 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.788+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fbcbc054b00 con 0x7fbcc4149ae0 2024-03-25T18:33:59.790 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.788+0000 7fbcba7fc640 10 client.4615.objecter ms_dispatch 0x7fbcc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:59.790 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.788+0000 7fbcba7fc640 3 client.4615.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:59.790 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.788+0000 7fbcba7fc640 3 client.4615.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:59.790 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.788+0000 7fbcba7fc640 20 client.4615.objecter dump_active .. 0 homeless 2024-03-25T18:33:59.790 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.789+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbcbc050610 con 0x7fbcc4149ae0 2024-03-25T18:33:59.790 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.789+0000 7fbcba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:59.790 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:33:59.789+0000 7fbcba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:00.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.029+0000 7fbccc1cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:00.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.029+0000 7fbccc1cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:00.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.029+0000 7fbccc1cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:34:00.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.029+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1826371649 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcc41d89f0 con 0x7fbcc4149ae0 2024-03-25T18:34:00.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.107+0000 7fbcba7fc640 1 -- 172.21.15.99:0/1826371649 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fbcbc052540 con 0x7fbcc4149ae0 2024-03-25T18:34:00.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.107+0000 7fbcba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:00.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.107+0000 7fbcba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:00.108 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:34:00.108 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:00.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.109+0000 7fbccc1cf640 10 librados: watch_flush enter 2024-03-25T18:34:00.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.109+0000 7fbccc1cf640 10 librados: watch_flush exit 2024-03-25T18:34:00.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.109+0000 7fbccc1cf640 20 client.4615.objecter shutdown clearing up homeless session... 2024-03-25T18:34:00.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.109+0000 7fbccc1cf640 10 client.4615.objecter successfully canceled tick 2024-03-25T18:34:00.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.109+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1826371649 >> v1:172.21.15.190:6800/4220409699 conn(0x7fbca806df00 legacy=0x7fbca80703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:00.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.109+0000 7fbccc1cf640 10 monclient: shutdown 2024-03-25T18:34:00.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.109+0000 7fbccc1cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:00.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.109+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1826371649 >> v1:172.21.15.190:6789/0 conn(0x7fbcc4149ae0 legacy=0x7fbcc4133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:00.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.109+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1826371649 shutdown_connections 2024-03-25T18:34:00.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.109+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1826371649 >> 172.21.15.99:0/1826371649 conn(0x7fbcc40b5d10 msgr2=0x7fbcc40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:00.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.109+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1826371649 shutdown_connections 2024-03-25T18:34:00.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.109+0000 7fbccc1cf640 1 -- 172.21.15.99:0/1826371649 wait complete. 2024-03-25T18:34:00.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:00.109+0000 7fbccc1cf640 1 librados: shutdown 2024-03-25T18:34:00.123 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:06.930 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:07.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.041+0000 7f068fe72640 10 monclient: get_monmap_and_config 2024-03-25T18:34:07.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.041+0000 7f068fe72640 10 monclient: build_initial_monmap 2024-03-25T18:34:07.042 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.041+0000 7f068fe72640 10 monclient: monmap: 2024-03-25T18:34:07.042 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:34:07.042 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:07.042 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:34:07.042283+0000 2024-03-25T18:34:07.042 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:34:07.042283+0000 2024-03-25T18:34:07.043 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:07.043 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:07.043 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:34:07.043 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:34:07.043 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:34:07.043 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:07.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.042+0000 7f068fe72640 1 Processor -- start 2024-03-25T18:34:07.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.042+0000 7f068fe72640 1 -- start start 2024-03-25T18:34:07.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.042+0000 7f068fe72640 10 monclient: init 2024-03-25T18:34:07.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.042+0000 7f068fe72640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:07.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.042+0000 7f068fe72640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:07.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.042+0000 7f068fe72640 10 monclient(hunting): picked mon.noname-a con 0x7f0688061830 addr v1:172.21.15.99:6789/0 2024-03-25T18:34:07.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.043+0000 7f068fe72640 10 monclient(hunting): picked mon.noname-b con 0x7f068813fcd0 addr v1:172.21.15.99:6790/0 2024-03-25T18:34:07.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.043+0000 7f068fe72640 10 monclient(hunting): picked mon.noname-c con 0x7f06880ae840 addr v1:172.21.15.190:6789/0 2024-03-25T18:34:07.044 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.043+0000 7f068fe72640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068806f760 con 0x7f0688061830 2024-03-25T18:34:07.045 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.043+0000 7f068fe72640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f068806fa40 con 0x7f06880ae840 2024-03-25T18:34:07.045 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.043+0000 7f068fe72640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06880ae0c0 con 0x7f068813fcd0 2024-03-25T18:34:07.045 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.043+0000 7f068fe72640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:07.045 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.043+0000 7f068fe72640 10 monclient(hunting): authenticate will time out at 7865.211140s 2024-03-25T18:34:07.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.055+0000 7f068e40f640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f06880ae840 0x7f0688145f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:54688/0 (socket says 172.21.15.99:54688) 2024-03-25T18:34:07.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.055+0000 7f068e40f640 1 -- 172.21.15.99:0/1424798100 learned_addr learned my addr 172.21.15.99:0/1424798100 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:34:07.075 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.074+0000 7f068cc0c640 1 -- 172.21.15.99:0/1424798100 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976644114 0 0) 0x7f068806fa40 con 0x7f06880ae840 2024-03-25T18:34:07.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.074+0000 7f068cc0c640 10 monclient(hunting): _negotiate 2024-03-25T18:34:07.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.074+0000 7f068cc0c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:07.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.074+0000 7f068cc0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:07.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.074+0000 7f068cc0c640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:34:07.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.074+0000 7f068cc0c640 1 -- 172.21.15.99:0/1424798100 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0674003520 con 0x7f06880ae840 2024-03-25T18:34:07.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 1 -- 172.21.15.99:0/1424798100 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 656383753 0 0) 0x7f0674003520 con 0x7f06880ae840 2024-03-25T18:34:07.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 1 -- 172.21.15.99:0/1424798100 >> v1:172.21.15.99:6790/0 conn(0x7f068813fcd0 legacy=0x7f0688145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:07.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 1 -- 172.21.15.99:0/1424798100 >> v1:172.21.15.99:6789/0 conn(0x7f0688061830 legacy=0x7f068813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:07.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 10 monclient: _finish_hunting 0 2024-03-25T18:34:07.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 1 monclient: found mon.noname-c 2024-03-25T18:34:07.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:07.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 1 -- 172.21.15.99:0/1424798100 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0688073040 con 0x7f06880ae840 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 10 monclient: _finish_auth 0 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 10 monclient: _check_auth_tickets 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 1 -- 172.21.15.99:0/1424798100 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0680001a10 con 0x7f06880ae840 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068fe72640 5 monclient: authenticate success, global_id 4618 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068fe72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068cc0c640 10 monclient: dump: 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:07.089 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:07.090 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:07.090 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:07.090 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:07.090 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:07.090 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:07.090 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.087+0000 7f068fe72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:07.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.102+0000 7f068cc0c640 1 -- 172.21.15.99:0/1424798100 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0680001b70 con 0x7f06880ae840 2024-03-25T18:34:07.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.102+0000 7f068cc0c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:07.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.102+0000 7f068cc0c640 1 -- 172.21.15.99:0/1424798100 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0680001f80 con 0x7f06880ae840 2024-03-25T18:34:07.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.102+0000 7f068fe72640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:07.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.102+0000 7f068fe72640 4 set_mon_vals no callback set 2024-03-25T18:34:07.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.102+0000 7f068cc0c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:07.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.102+0000 7f068cc0c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:07.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.102+0000 7f068cc0c640 10 monclient: dump: 2024-03-25T18:34:07.104 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:07.104 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:07.104 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:07.104 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:07.105 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:07.105 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:07.105 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:07.105 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:07.105 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:07.105 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:07.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.103+0000 7f068fe72640 10 monclient: shutdown 2024-03-25T18:34:07.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.103+0000 7f068fe72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:07.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.103+0000 7f068fe72640 1 -- 172.21.15.99:0/1424798100 >> v1:172.21.15.190:6789/0 conn(0x7f06880ae840 legacy=0x7f0688145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:07.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.103+0000 7f068fe72640 1 -- 172.21.15.99:0/1424798100 shutdown_connections 2024-03-25T18:34:07.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.103+0000 7f068fe72640 1 -- 172.21.15.99:0/1424798100 >> 172.21.15.99:0/1424798100 conn(0x7f06880b0ef0 msgr2=0x7f06880b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:07.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.104+0000 7f068fe72640 1 -- 172.21.15.99:0/1424798100 shutdown_connections 2024-03-25T18:34:07.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.104+0000 7f068fe72640 1 -- 172.21.15.99:0/1424798100 wait complete. 2024-03-25T18:34:07.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.104+0000 7f068fe72640 10 monclient: build_initial_monmap 2024-03-25T18:34:07.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.104+0000 7f068fe72640 10 monclient: monmap: 2024-03-25T18:34:07.107 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:34:07.107 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:07.107 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.105+0000 7f068fe72640 1 librados: starting msgr at 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.105+0000 7f068fe72640 1 librados: starting objecter 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.105+0000 7f068fe72640 1 Processor -- start 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.105+0000 7f068fe72640 1 -- start start 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.105+0000 7f068fe72640 1 librados: setting wanted keys 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.105+0000 7f068fe72640 1 librados: calling monclient init 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.105+0000 7f068fe72640 10 monclient: init 2024-03-25T18:34:07.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.105+0000 7f068fe72640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:07.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.105+0000 7f068fe72640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:34:07.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.105+0000 7f068fe72640 10 monclient(hunting): picked mon.noname-b con 0x7f0688061830 addr v1:172.21.15.190:6789/0 2024-03-25T18:34:07.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.105+0000 7f068fe72640 10 monclient(hunting): picked mon.noname-a con 0x7f06880ae840 addr v1:172.21.15.99:6789/0 2024-03-25T18:34:07.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.106+0000 7f068fe72640 10 monclient(hunting): picked mon.noname-c con 0x7f068813fcd0 addr v1:172.21.15.99:6790/0 2024-03-25T18:34:07.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.106+0000 7f068fe72640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0688137eb0 con 0x7f06880ae840 2024-03-25T18:34:07.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.106+0000 7f068fe72640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0688138a60 con 0x7f0688061830 2024-03-25T18:34:07.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.106+0000 7f068fe72640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0688138d40 con 0x7f068813fcd0 2024-03-25T18:34:07.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.106+0000 7f068fe72640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:07.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.106+0000 7f068fe72640 10 monclient(hunting): authenticate will time out at 7865.273996s 2024-03-25T18:34:07.121 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.120+0000 7f068e40f640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f068813fcd0 0x7f0688137430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:45574/0 (socket says 172.21.15.99:45574) 2024-03-25T18:34:07.121 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.120+0000 7f068e40f640 1 -- 172.21.15.99:0/1412541418 learned_addr learned my addr 172.21.15.99:0/1412541418 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:34:07.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.122+0000 7f067e7fc640 10 client.?.objecter ms_handle_connect 0x7f068813fcd0 2024-03-25T18:34:07.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.122+0000 7f067e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:07.132 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.131+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664241281 0 0) 0x7f0688138d40 con 0x7f068813fcd0 2024-03-25T18:34:07.132 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.131+0000 7f067e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:07.132 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.131+0000 7f067e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:07.132 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.131+0000 7f067e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:07.132 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.131+0000 7f067e7fc640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:34:07.132 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.131+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f066c004eb0 con 0x7f068813fcd0 2024-03-25T18:34:07.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.133+0000 7f067e7fc640 10 client.?.objecter ms_handle_connect 0x7f06880ae840 2024-03-25T18:34:07.135 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.133+0000 7f067e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:07.137 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.136+0000 7f067e7fc640 10 client.?.objecter ms_handle_connect 0x7f0688061830 2024-03-25T18:34:07.137 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.136+0000 7f067e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:07.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.142+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4055370705 0 0) 0x7f066c004eb0 con 0x7f068813fcd0 2024-03-25T18:34:07.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.142+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0688138d40 con 0x7f068813fcd0 2024-03-25T18:34:07.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.143+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0680002000 con 0x7f068813fcd0 2024-03-25T18:34:07.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.143+0000 7f067e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:07.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.143+0000 7f067e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:07.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.143+0000 7f067e7fc640 10 monclient(hunting): dump: 2024-03-25T18:34:07.145 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:07.145 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:07.145 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:07.145 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:07.145 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:07.145 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:07.145 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:07.145 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:07.145 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:07.145 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:07.145 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.143+0000 7f067e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:07.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.145+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136732071 0 0) 0x7f0688137eb0 con 0x7f06880ae840 2024-03-25T18:34:07.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.145+0000 7f067e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:07.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.145+0000 7f067e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:07.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.145+0000 7f067e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:07.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.145+0000 7f067e7fc640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:34:07.146 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.145+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f066c004eb0 con 0x7f06880ae840 2024-03-25T18:34:07.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.148+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544029300 0 0) 0x7f0688138a60 con 0x7f0688061830 2024-03-25T18:34:07.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.148+0000 7f067e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:07.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.148+0000 7f067e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:07.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.148+0000 7f067e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:07.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.148+0000 7f067e7fc640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:34:07.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.148+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0688137eb0 con 0x7f0688061830 2024-03-25T18:34:07.153 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.152+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3398451887 0 0) 0x7f066c004eb0 con 0x7f06880ae840 2024-03-25T18:34:07.153 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.152+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0688138a60 con 0x7f06880ae840 2024-03-25T18:34:07.153 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.152+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0678001830 con 0x7f06880ae840 2024-03-25T18:34:07.153 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.152+0000 7f067e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:07.153 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.152+0000 7f067e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:07.153 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.152+0000 7f067e7fc640 10 monclient(hunting): dump: 2024-03-25T18:34:07.154 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:07.154 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:07.154 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:07.154 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:07.154 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:07.154 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:07.154 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:07.154 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:07.154 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:07.154 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:07.158 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 61304611 0 0) 0x7f0688138d40 con 0x7f068813fcd0 2024-03-25T18:34:07.158 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 >> v1:172.21.15.190:6789/0 conn(0x7f0688061830 legacy=0x7f06881396c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:07.158 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 >> v1:172.21.15.99:6789/0 conn(0x7f06880ae840 legacy=0x7f06881358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:07.158 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f067e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:07.158 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f067e7fc640 1 monclient: found mon.c 2024-03-25T18:34:07.158 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f067e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:07.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f067e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:07.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0688137b40 con 0x7f068813fcd0 2024-03-25T18:34:07.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f067e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:07.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f067e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:07.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f067e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:07.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f068fe72640 5 monclient: authenticate success, global_id 4616 2024-03-25T18:34:07.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f068fe72640 10 monclient: _renew_subs 2024-03-25T18:34:07.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f068fe72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:07.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f068fe72640 1 -- 172.21.15.99:0/1412541418 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06881369c0 con 0x7f068813fcd0 2024-03-25T18:34:07.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f068fe72640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:07.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f068fe72640 10 monclient: _renew_subs 2024-03-25T18:34:07.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f068fe72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:07.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f068fe72640 1 -- 172.21.15.99:0/1412541418 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06881e3160 con 0x7f068813fcd0 2024-03-25T18:34:07.159 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.157+0000 7f068fe72640 1 librados: init done 2024-03-25T18:34:07.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.158+0000 7f068fe72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:07.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.158+0000 7f068fe72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:07.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.158+0000 7f068fe72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:07.160 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.158+0000 7f068fe72640 1 -- 172.21.15.99:0/1412541418 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0688135690 con 0x7f068813fcd0 2024-03-25T18:34:07.457 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.456+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0680004e10 con 0x7f068813fcd0 2024-03-25T18:34:07.458 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.456+0000 7f067e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:07.458 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.456+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0680005120 con 0x7f068813fcd0 2024-03-25T18:34:07.458 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.456+0000 7f067e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:07.458 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.456+0000 7f067e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:07.458 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.456+0000 7f067e7fc640 10 monclient: dump: 2024-03-25T18:34:07.458 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:07.458 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:07.458 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:07.458 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:07.458 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:07.458 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:07.458 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:07.458 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:07.458 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:07.459 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:07.459 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.456+0000 7f067ffff640 4 set_mon_vals no callback set 2024-03-25T18:34:07.459 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.457+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f068001b0f0 con 0x7f068813fcd0 2024-03-25T18:34:07.459 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.458+0000 7f068dc0e640 10 monclient: get_auth_request con 0x7f066c06df70 auth_method 0 2024-03-25T18:34:07.459 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.458+0000 7f067e7fc640 10 client.4616.objecter ms_handle_connect 0x7f066c06df70 2024-03-25T18:34:07.464 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.463+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f0680054d20 con 0x7f068813fcd0 2024-03-25T18:34:07.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.463+0000 7f067e7fc640 10 client.4616.objecter ms_dispatch 0x7f0688000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:07.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.463+0000 7f067e7fc640 3 client.4616.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:07.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.463+0000 7f067e7fc640 3 client.4616.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:07.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.463+0000 7f067e7fc640 20 client.4616.objecter dump_active .. 0 homeless 2024-03-25T18:34:07.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.463+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0680050660 con 0x7f068813fcd0 2024-03-25T18:34:07.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.463+0000 7f067e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:07.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.463+0000 7f067e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:07.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.716+0000 7f068fe72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:07.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.716+0000 7f068fe72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:07.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.716+0000 7f068fe72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:07.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.716+0000 7f068fe72640 1 -- 172.21.15.99:0/1412541418 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06880627b0 con 0x7f068813fcd0 2024-03-25T18:34:07.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.752+0000 7f067e7fc640 1 -- 172.21.15.99:0/1412541418 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f0680052760 con 0x7f068813fcd0 2024-03-25T18:34:07.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.753+0000 7f067e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:07.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.753+0000 7f067e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:07.754 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:34:07.754 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:07.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.755+0000 7f068fe72640 10 librados: watch_flush enter 2024-03-25T18:34:07.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.755+0000 7f068fe72640 10 librados: watch_flush exit 2024-03-25T18:34:07.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.755+0000 7f068fe72640 20 client.4616.objecter shutdown clearing up homeless session... 2024-03-25T18:34:07.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.755+0000 7f068fe72640 10 client.4616.objecter successfully canceled tick 2024-03-25T18:34:07.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.755+0000 7f068fe72640 1 -- 172.21.15.99:0/1412541418 >> v1:172.21.15.190:6800/4220409699 conn(0x7f066c06df70 legacy=0x7f066c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:07.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.755+0000 7f068fe72640 10 monclient: shutdown 2024-03-25T18:34:07.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.755+0000 7f068fe72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:07.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.755+0000 7f068fe72640 1 -- 172.21.15.99:0/1412541418 >> v1:172.21.15.99:6790/0 conn(0x7f068813fcd0 legacy=0x7f0688137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:07.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.755+0000 7f068fe72640 1 -- 172.21.15.99:0/1412541418 shutdown_connections 2024-03-25T18:34:07.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.755+0000 7f068fe72640 1 -- 172.21.15.99:0/1412541418 >> 172.21.15.99:0/1412541418 conn(0x7f06880b0ef0 msgr2=0x7f06880afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:07.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.756+0000 7f068fe72640 1 -- 172.21.15.99:0/1412541418 shutdown_connections 2024-03-25T18:34:07.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.756+0000 7f068fe72640 1 -- 172.21.15.99:0/1412541418 wait complete. 2024-03-25T18:34:07.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:07.756+0000 7f068fe72640 1 librados: shutdown 2024-03-25T18:34:07.769 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:14.575 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:14.693 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.692+0000 7fe06884e640 10 monclient: get_monmap_and_config 2024-03-25T18:34:14.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.692+0000 7fe06884e640 10 monclient: build_initial_monmap 2024-03-25T18:34:14.694 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.692+0000 7fe06884e640 10 monclient: monmap: 2024-03-25T18:34:14.694 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:34:14.694 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:14.694 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:34:14.693895+0000 2024-03-25T18:34:14.694 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:34:14.693895+0000 2024-03-25T18:34:14.694 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:14.694 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:14.694 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:34:14.694 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:34:14.694 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:34:14.695 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:14.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.693+0000 7fe06884e640 1 Processor -- start 2024-03-25T18:34:14.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.693+0000 7fe06884e640 1 -- start start 2024-03-25T18:34:14.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.693+0000 7fe06884e640 10 monclient: init 2024-03-25T18:34:14.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.694+0000 7fe06884e640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:14.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.694+0000 7fe06884e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:14.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.694+0000 7fe06884e640 10 monclient(hunting): picked mon.noname-c con 0x7fe0640a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:34:14.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.694+0000 7fe06884e640 10 monclient(hunting): picked mon.noname-b con 0x7fe06413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:34:14.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.694+0000 7fe06884e640 10 monclient(hunting): picked mon.noname-a con 0x7fe0641430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:34:14.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.694+0000 7fe06884e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe064142520 con 0x7fe0641430c0 2024-03-25T18:34:14.696 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.694+0000 7fe06884e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640b28a0 con 0x7fe0640a98f0 2024-03-25T18:34:14.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.694+0000 7fe06884e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640b2a30 con 0x7fe06413f710 2024-03-25T18:34:14.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.694+0000 7fe06884e640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:14.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.694+0000 7fe06884e640 10 monclient(hunting): authenticate will time out at 7872.862365s 2024-03-25T18:34:14.704 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.703+0000 7fe06259c640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe0640a98f0 0x7fe0640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:41822/0 (socket says 172.21.15.99:41822) 2024-03-25T18:34:14.704 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.703+0000 7fe06259c640 1 -- 172.21.15.99:0/1908638612 learned_addr learned my addr 172.21.15.99:0/1908638612 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:34:14.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.711+0000 7fe061d9b640 1 -- 172.21.15.99:0/1908638612 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057101074 0 0) 0x7fe0640b28a0 con 0x7fe0640a98f0 2024-03-25T18:34:14.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.711+0000 7fe061d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:14.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.711+0000 7fe061d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:14.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.711+0000 7fe061d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:14.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.711+0000 7fe061d9b640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:34:14.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.711+0000 7fe061d9b640 1 -- 172.21.15.99:0/1908638612 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe050003520 con 0x7fe0640a98f0 2024-03-25T18:34:14.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.712+0000 7fe061d9b640 1 -- 172.21.15.99:0/1908638612 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914450403 0 0) 0x7fe064142520 con 0x7fe0641430c0 2024-03-25T18:34:14.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.712+0000 7fe061d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:14.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.712+0000 7fe061d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:14.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.712+0000 7fe061d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:14.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.712+0000 7fe061d9b640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:34:14.713 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.712+0000 7fe061d9b640 1 -- 172.21.15.99:0/1908638612 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0640b28a0 con 0x7fe0641430c0 2024-03-25T18:34:14.722 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 1 -- 172.21.15.99:0/1908638612 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2852526220 0 0) 0x7fe050003520 con 0x7fe0640a98f0 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 1 -- 172.21.15.99:0/1908638612 >> v1:172.21.15.99:6790/0 conn(0x7fe06413f710 legacy=0x7fe0641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 1 -- 172.21.15.99:0/1908638612 >> v1:172.21.15.99:6789/0 conn(0x7fe0641430c0 legacy=0x7fe06414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 1 monclient: found mon.noname-c 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 1 -- 172.21.15.99:0/1908638612 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0640b44c0 con 0x7fe0640a98f0 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 10 monclient: _finish_auth 0 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 1 -- 172.21.15.99:0/1908638612 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe054001690 con 0x7fe0640a98f0 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe061d9b640 10 monclient: dump: 2024-03-25T18:34:14.723 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:14.724 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:14.724 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:14.724 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:14.724 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:14.724 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:14.724 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:14.724 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:14.724 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:14.724 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:14.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe06884e640 5 monclient: authenticate success, global_id 4624 2024-03-25T18:34:14.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.721+0000 7fe06884e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:14.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.727+0000 7fe061d9b640 1 -- 172.21.15.99:0/1908638612 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0540017f0 con 0x7fe0640a98f0 2024-03-25T18:34:14.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.727+0000 7fe061d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:14.728 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.727+0000 7fe061d9b640 1 -- 172.21.15.99:0/1908638612 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe054001c00 con 0x7fe0640a98f0 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.727+0000 7fe061d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.727+0000 7fe061d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.727+0000 7fe061d9b640 10 monclient: dump: 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.727+0000 7fe06884e640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:14.729 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.727+0000 7fe06884e640 4 set_mon_vals no callback set 2024-03-25T18:34:14.731 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.728+0000 7fe06884e640 10 monclient: shutdown 2024-03-25T18:34:14.731 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.728+0000 7fe06884e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:14.731 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.728+0000 7fe06884e640 1 -- 172.21.15.99:0/1908638612 >> v1:172.21.15.190:6789/0 conn(0x7fe0640a98f0 legacy=0x7fe0640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:14.731 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.728+0000 7fe06884e640 1 -- 172.21.15.99:0/1908638612 shutdown_connections 2024-03-25T18:34:14.731 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.728+0000 7fe06884e640 1 -- 172.21.15.99:0/1908638612 >> 172.21.15.99:0/1908638612 conn(0x7fe0640b7ea0 msgr2=0x7fe0640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:14.731 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.728+0000 7fe06884e640 1 -- 172.21.15.99:0/1908638612 shutdown_connections 2024-03-25T18:34:14.731 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.728+0000 7fe06884e640 1 -- 172.21.15.99:0/1908638612 wait complete. 2024-03-25T18:34:14.731 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.729+0000 7fe06884e640 10 monclient: build_initial_monmap 2024-03-25T18:34:14.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.729+0000 7fe06884e640 10 monclient: monmap: 2024-03-25T18:34:14.732 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:34:14.732 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:14.732 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:34:14.732 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:34:14.732 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:14.732 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:34:14.732 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:34:14.732 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:34:14.732 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:34:14.732 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:14.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.729+0000 7fe06884e640 1 librados: starting msgr at 2024-03-25T18:34:14.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.729+0000 7fe06884e640 1 librados: starting objecter 2024-03-25T18:34:14.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.730+0000 7fe06884e640 1 Processor -- start 2024-03-25T18:34:14.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.730+0000 7fe06884e640 1 -- start start 2024-03-25T18:34:14.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.730+0000 7fe06884e640 1 librados: setting wanted keys 2024-03-25T18:34:14.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.730+0000 7fe06884e640 1 librados: calling monclient init 2024-03-25T18:34:14.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.730+0000 7fe06884e640 10 monclient: init 2024-03-25T18:34:14.734 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.731+0000 7fe06884e640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:14.734 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.731+0000 7fe06884e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:14.734 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.731+0000 7fe06884e640 10 monclient(hunting): picked mon.noname-a con 0x7fe0640a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:34:14.734 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.731+0000 7fe06884e640 10 monclient(hunting): picked mon.noname-b con 0x7fe06413f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:34:14.734 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.731+0000 7fe06884e640 10 monclient(hunting): picked mon.noname-c con 0x7fe0641430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:34:14.734 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.731+0000 7fe06884e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe064139260 con 0x7fe0640a98f0 2024-03-25T18:34:14.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.731+0000 7fe06884e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640ac1f0 con 0x7fe06413f710 2024-03-25T18:34:14.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.731+0000 7fe06884e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0641398a0 con 0x7fe0641430c0 2024-03-25T18:34:14.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.731+0000 7fe06884e640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:14.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.731+0000 7fe06884e640 10 monclient(hunting): authenticate will time out at 7872.899310s 2024-03-25T18:34:14.738 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.737+0000 7fe062d9d640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7fe0641430c0 0x7fe0640ac610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:48940/0 (socket says 172.21.15.99:48940) 2024-03-25T18:34:14.738 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.737+0000 7fe062d9d640 1 -- 172.21.15.99:0/86030729 learned_addr learned my addr 172.21.15.99:0/86030729 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:34:14.738 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.738+0000 7fe05b7fe640 10 client.?.objecter ms_handle_connect 0x7fe0641430c0 2024-03-25T18:34:14.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.738+0000 7fe05b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:14.741 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.740+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926397697 0 0) 0x7fe0641398a0 con 0x7fe0641430c0 2024-03-25T18:34:14.741 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.740+0000 7fe05b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:14.741 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.740+0000 7fe05b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:14.741 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.740+0000 7fe05b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:14.741 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.740+0000 7fe05b7fe640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:34:14.741 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.740+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe044004eb0 con 0x7fe0641430c0 2024-03-25T18:34:14.749 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.747+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2888293938 0 0) 0x7fe044004eb0 con 0x7fe0641430c0 2024-03-25T18:34:14.749 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.747+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0641398a0 con 0x7fe0641430c0 2024-03-25T18:34:14.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.751+0000 7fe05b7fe640 10 client.?.objecter ms_handle_connect 0x7fe06413f710 2024-03-25T18:34:14.752 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.751+0000 7fe05b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:14.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.752+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe05c001df0 con 0x7fe0641430c0 2024-03-25T18:34:14.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.752+0000 7fe05b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:14.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.752+0000 7fe05b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:14.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.752+0000 7fe05b7fe640 10 monclient(hunting): dump: 2024-03-25T18:34:14.753 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:14.753 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:14.753 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:14.753 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:14.753 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:14.753 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:14.753 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:14.754 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:14.754 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:14.754 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:14.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.752+0000 7fe05b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:14.755 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.754+0000 7fe05b7fe640 10 client.?.objecter ms_handle_connect 0x7fe0640a98f0 2024-03-25T18:34:14.755 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.754+0000 7fe05b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:14.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.762+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154867303 0 0) 0x7fe0640ac1f0 con 0x7fe06413f710 2024-03-25T18:34:14.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.762+0000 7fe05b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:14.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.762+0000 7fe05b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:14.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.762+0000 7fe05b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:14.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.762+0000 7fe05b7fe640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:34:14.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.762+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe044004eb0 con 0x7fe06413f710 2024-03-25T18:34:14.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276059997 0 0) 0x7fe064139260 con 0x7fe0640a98f0 2024-03-25T18:34:14.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:14.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:14.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:14.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:34:14.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0640ac1f0 con 0x7fe0640a98f0 2024-03-25T18:34:14.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1437270977 0 0) 0x7fe0641398a0 con 0x7fe0641430c0 2024-03-25T18:34:14.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 >> v1:172.21.15.190:6789/0 conn(0x7fe06413f710 legacy=0x7fe064136640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:14.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 >> v1:172.21.15.99:6789/0 conn(0x7fe0640a98f0 legacy=0x7fe064138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:14.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:14.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 1 monclient: found mon.c 2024-03-25T18:34:14.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:14.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:14.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe064139a30 con 0x7fe0641430c0 2024-03-25T18:34:14.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:14.767 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:14.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe05b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:14.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe06884e640 5 monclient: authenticate success, global_id 4622 2024-03-25T18:34:14.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.764+0000 7fe06884e640 10 monclient: _renew_subs 2024-03-25T18:34:14.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.765+0000 7fe06884e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:14.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.765+0000 7fe06884e640 1 -- 172.21.15.99:0/86030729 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe064137e80 con 0x7fe0641430c0 2024-03-25T18:34:14.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.765+0000 7fe06884e640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:14.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.765+0000 7fe06884e640 10 monclient: _renew_subs 2024-03-25T18:34:14.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.765+0000 7fe06884e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:14.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.765+0000 7fe06884e640 1 -- 172.21.15.99:0/86030729 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0641e3690 con 0x7fe0641430c0 2024-03-25T18:34:14.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.765+0000 7fe06884e640 1 librados: init done 2024-03-25T18:34:14.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.766+0000 7fe06884e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:14.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.766+0000 7fe06884e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:14.768 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.766+0000 7fe06884e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:14.769 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:14.766+0000 7fe06884e640 1 -- 172.21.15.99:0/86030729 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0640627b0 con 0x7fe0641430c0 2024-03-25T18:34:15.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.058+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe05c001f50 con 0x7fe0641430c0 2024-03-25T18:34:15.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.058+0000 7fe05b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:15.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.058+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fe05c003de0 con 0x7fe0641430c0 2024-03-25T18:34:15.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.058+0000 7fe05b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:15.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.058+0000 7fe05b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:15.060 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.058+0000 7fe05b7fe640 10 monclient: dump: 2024-03-25T18:34:15.060 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:15.060 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:15.060 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:15.060 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:15.060 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:15.060 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:15.061 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:15.061 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:15.061 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:15.061 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:15.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.058+0000 7fe06159a640 4 set_mon_vals no callback set 2024-03-25T18:34:15.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.059+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fe05c004000 con 0x7fe0641430c0 2024-03-25T18:34:15.061 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.060+0000 7fe06259c640 10 monclient: get_auth_request con 0x7fe04406de00 auth_method 0 2024-03-25T18:34:15.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.060+0000 7fe05b7fe640 10 client.4622.objecter ms_handle_connect 0x7fe04406de00 2024-03-25T18:34:15.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.066+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fe05c054b80 con 0x7fe0641430c0 2024-03-25T18:34:15.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.066+0000 7fe05b7fe640 10 client.4622.objecter ms_dispatch 0x7fe064000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:15.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.066+0000 7fe05b7fe640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:15.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.066+0000 7fe05b7fe640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:15.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.066+0000 7fe05b7fe640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-25T18:34:15.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.066+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe05c050690 con 0x7fe0641430c0 2024-03-25T18:34:15.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.066+0000 7fe05b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:15.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.066+0000 7fe05b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:15.195 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:09:15.788 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.247 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:09:15.839 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.275 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:09:15.868 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.291 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:09:15.850 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.299 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.298+0000 7fe06884e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:15.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.298+0000 7fe06884e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:15.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.298+0000 7fe06884e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:15.300 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.298+0000 7fe06884e640 1 -- 172.21.15.99:0/86030729 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe06413df50 con 0x7fe0641430c0 2024-03-25T18:34:15.301 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:09:15.860 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.314 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:09:15.906 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.325 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:09:15.883 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.347 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:09:15.940 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:15.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.354+0000 7fe05b7fe640 1 -- 172.21.15.99:0/86030729 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fe05c0525c0 con 0x7fe0641430c0 2024-03-25T18:34:15.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.354+0000 7fe05b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:15.355 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.354+0000 7fe05b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:15.355 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:34:15.355 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:15.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.356+0000 7fe06884e640 10 librados: watch_flush enter 2024-03-25T18:34:15.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.356+0000 7fe06884e640 10 librados: watch_flush exit 2024-03-25T18:34:15.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.356+0000 7fe06884e640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-25T18:34:15.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.356+0000 7fe06884e640 10 client.4622.objecter successfully canceled tick 2024-03-25T18:34:15.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.356+0000 7fe06884e640 1 -- 172.21.15.99:0/86030729 >> v1:172.21.15.190:6800/4220409699 conn(0x7fe04406de00 legacy=0x7fe0440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:15.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.356+0000 7fe06884e640 10 monclient: shutdown 2024-03-25T18:34:15.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.356+0000 7fe06884e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:15.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.356+0000 7fe06884e640 1 -- 172.21.15.99:0/86030729 >> v1:172.21.15.99:6790/0 conn(0x7fe0641430c0 legacy=0x7fe0640ac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:15.358 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.356+0000 7fe06884e640 1 -- 172.21.15.99:0/86030729 shutdown_connections 2024-03-25T18:34:15.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.356+0000 7fe06884e640 1 -- 172.21.15.99:0/86030729 >> 172.21.15.99:0/86030729 conn(0x7fe0640b7ea0 msgr2=0x7fe064074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:15.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.356+0000 7fe06884e640 1 -- 172.21.15.99:0/86030729 shutdown_connections 2024-03-25T18:34:15.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.356+0000 7fe06884e640 1 -- 172.21.15.99:0/86030729 wait complete. 2024-03-25T18:34:15.359 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:15.356+0000 7fe06884e640 1 librados: shutdown 2024-03-25T18:34:15.369 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:15.373 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:09:15.966 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:22.176 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:22.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.292+0000 7f304aa93640 10 monclient: get_monmap_and_config 2024-03-25T18:34:22.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.292+0000 7f304aa93640 10 monclient: build_initial_monmap 2024-03-25T18:34:22.294 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.293+0000 7f304aa93640 10 monclient: monmap: 2024-03-25T18:34:22.294 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:34:22.294 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:22.294 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:34:22.294034+0000 2024-03-25T18:34:22.294 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:34:22.294034+0000 2024-03-25T18:34:22.294 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:22.295 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:22.295 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:34:22.295 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:34:22.295 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:34:22.295 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:22.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.293+0000 7f304aa93640 1 Processor -- start 2024-03-25T18:34:22.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.293+0000 7f304aa93640 1 -- start start 2024-03-25T18:34:22.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.293+0000 7f304aa93640 10 monclient: init 2024-03-25T18:34:22.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.294+0000 7f304aa93640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:22.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.294+0000 7f304aa93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:22.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.294+0000 7f304aa93640 10 monclient(hunting): picked mon.noname-a con 0x7f30440b4d60 addr v1:172.21.15.99:6789/0 2024-03-25T18:34:22.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.294+0000 7f304aa93640 10 monclient(hunting): picked mon.noname-c con 0x7f30440b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:34:22.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.294+0000 7f304aa93640 10 monclient(hunting): picked mon.noname-b con 0x7f304414be90 addr v1:172.21.15.99:6790/0 2024-03-25T18:34:22.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.294+0000 7f304aa93640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30440af940 con 0x7f30440b4d60 2024-03-25T18:34:22.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.294+0000 7f304aa93640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30440b0a70 con 0x7f30440b1730 2024-03-25T18:34:22.296 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.294+0000 7f304aa93640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30440b0d50 con 0x7f304414be90 2024-03-25T18:34:22.297 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.294+0000 7f304aa93640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:22.297 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.294+0000 7f304aa93640 10 monclient(hunting): authenticate will time out at 7880.462336s 2024-03-25T18:34:22.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.310+0000 7f3043fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f30440b1730 0x7f30440b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:46350/0 (socket says 172.21.15.99:46350) 2024-03-25T18:34:22.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.310+0000 7f3043fff640 1 -- 172.21.15.99:0/2601558465 learned_addr learned my addr 172.21.15.99:0/2601558465 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:34:22.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.317+0000 7f30437fe640 1 -- 172.21.15.99:0/2601558465 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221546707 0 0) 0x7f30440b0a70 con 0x7f30440b1730 2024-03-25T18:34:22.318 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.317+0000 7f30437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:22.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.317+0000 7f30437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:22.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.317+0000 7f30437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:22.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.317+0000 7f30437fe640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:34:22.319 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.317+0000 7f30437fe640 1 -- 172.21.15.99:0/2601558465 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3034003520 con 0x7f30440b1730 2024-03-25T18:34:22.322 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.321+0000 7f30437fe640 1 -- 172.21.15.99:0/2601558465 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100414218 0 0) 0x7f30440b0d50 con 0x7f304414be90 2024-03-25T18:34:22.322 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.321+0000 7f30437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:22.322 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.321+0000 7f30437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:22.322 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.321+0000 7f30437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:22.322 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.321+0000 7f30437fe640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:34:22.322 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.321+0000 7f30437fe640 1 -- 172.21.15.99:0/2601558465 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30440b0a70 con 0x7f304414be90 2024-03-25T18:34:22.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 1 -- 172.21.15.99:0/2601558465 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1578480401 0 0) 0x7f3034003520 con 0x7f30440b1730 2024-03-25T18:34:22.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 1 -- 172.21.15.99:0/2601558465 >> v1:172.21.15.99:6790/0 conn(0x7f304414be90 legacy=0x7f304414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:22.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 1 -- 172.21.15.99:0/2601558465 >> v1:172.21.15.99:6789/0 conn(0x7f30440b4d60 legacy=0x7f304413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:22.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:22.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 1 monclient: found mon.noname-c 2024-03-25T18:34:22.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:22.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:34:22.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 1 -- 172.21.15.99:0/2601558465 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30440b1030 con 0x7f30440b1730 2024-03-25T18:34:22.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:22.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:22.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:22.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 1 -- 172.21.15.99:0/2601558465 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f3030001690 con 0x7f30440b1730 2024-03-25T18:34:22.327 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f304aa93640 5 monclient: authenticate success, global_id 4630 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f304aa93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f30437fe640 10 monclient: dump: 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:22.328 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.326+0000 7f304aa93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:22.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.347+0000 7f30437fe640 1 -- 172.21.15.99:0/2601558465 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30300017f0 con 0x7f30440b1730 2024-03-25T18:34:22.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.347+0000 7f30437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:22.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.347+0000 7f30437fe640 1 -- 172.21.15.99:0/2601558465 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f3030001c00 con 0x7f30440b1730 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.347+0000 7f304aa93640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.347+0000 7f304aa93640 4 set_mon_vals no callback set 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.347+0000 7f30437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.347+0000 7f30437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.347+0000 7f30437fe640 10 monclient: dump: 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:22.349 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:22.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.348+0000 7f304aa93640 10 monclient: shutdown 2024-03-25T18:34:22.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.348+0000 7f304aa93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:22.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.348+0000 7f304aa93640 1 -- 172.21.15.99:0/2601558465 >> v1:172.21.15.190:6789/0 conn(0x7f30440b1730 legacy=0x7f30440b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:22.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.348+0000 7f304aa93640 1 -- 172.21.15.99:0/2601558465 shutdown_connections 2024-03-25T18:34:22.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.348+0000 7f304aa93640 1 -- 172.21.15.99:0/2601558465 >> 172.21.15.99:0/2601558465 conn(0x7f30440617a0 msgr2=0x7f3044061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:22.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.348+0000 7f304aa93640 1 -- 172.21.15.99:0/2601558465 shutdown_connections 2024-03-25T18:34:22.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.348+0000 7f304aa93640 1 -- 172.21.15.99:0/2601558465 wait complete. 2024-03-25T18:34:22.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.349+0000 7f304aa93640 10 monclient: build_initial_monmap 2024-03-25T18:34:22.351 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.349+0000 7f304aa93640 10 monclient: monmap: 2024-03-25T18:34:22.351 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:34:22.351 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:22.351 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:34:22.351 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:34:22.351 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:22.351 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:34:22.352 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:34:22.352 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:34:22.352 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:34:22.352 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:22.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.350+0000 7f304aa93640 1 librados: starting msgr at 2024-03-25T18:34:22.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.350+0000 7f304aa93640 1 librados: starting objecter 2024-03-25T18:34:22.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.350+0000 7f304aa93640 1 Processor -- start 2024-03-25T18:34:22.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.350+0000 7f304aa93640 1 -- start start 2024-03-25T18:34:22.353 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.350+0000 7f304aa93640 1 librados: setting wanted keys 2024-03-25T18:34:22.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.350+0000 7f304aa93640 1 librados: calling monclient init 2024-03-25T18:34:22.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.350+0000 7f304aa93640 10 monclient: init 2024-03-25T18:34:22.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.351+0000 7f304aa93640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:22.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.351+0000 7f304aa93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:34:22.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.351+0000 7f304aa93640 10 monclient(hunting): picked mon.noname-b con 0x7f30440b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:34:22.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.351+0000 7f304aa93640 10 monclient(hunting): picked mon.noname-a con 0x7f30440b4d60 addr v1:172.21.15.99:6789/0 2024-03-25T18:34:22.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.351+0000 7f304aa93640 10 monclient(hunting): picked mon.noname-c con 0x7f304414be90 addr v1:172.21.15.99:6790/0 2024-03-25T18:34:22.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.351+0000 7f304aa93640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304413bd10 con 0x7f30440b4d60 2024-03-25T18:34:22.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.352+0000 7f304aa93640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3044141c30 con 0x7f30440b1730 2024-03-25T18:34:22.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.352+0000 7f304aa93640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3044141f10 con 0x7f304414be90 2024-03-25T18:34:22.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.352+0000 7f304aa93640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:22.354 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.352+0000 7f304aa93640 10 monclient(hunting): authenticate will time out at 7880.519944s 2024-03-25T18:34:22.363 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.362+0000 7f3049030640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f304414be90 0x7f30441e35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:53524/0 (socket says 172.21.15.99:53524) 2024-03-25T18:34:22.363 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.362+0000 7f3049030640 1 -- 172.21.15.99:0/4254030341 learned_addr learned my addr 172.21.15.99:0/4254030341 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:34:22.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.364+0000 7f30417fa640 10 client.?.objecter ms_handle_connect 0x7f304414be90 2024-03-25T18:34:22.365 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.364+0000 7f30417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:22.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.366+0000 7f30417fa640 10 client.?.objecter ms_handle_connect 0x7f30440b4d60 2024-03-25T18:34:22.367 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.366+0000 7f30417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:22.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.372+0000 7f30417fa640 10 client.?.objecter ms_handle_connect 0x7f30440b1730 2024-03-25T18:34:22.373 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.372+0000 7f30417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:22.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.373+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508392344 0 0) 0x7f3044141f10 con 0x7f304414be90 2024-03-25T18:34:22.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.373+0000 7f30417fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:22.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.373+0000 7f30417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:22.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.373+0000 7f30417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:22.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.373+0000 7f30417fa640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:34:22.374 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.373+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3028004eb0 con 0x7f304414be90 2024-03-25T18:34:22.377 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.376+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301328601 0 0) 0x7f304413bd10 con 0x7f30440b4d60 2024-03-25T18:34:22.377 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.376+0000 7f30417fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:22.377 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.376+0000 7f30417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:22.377 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.376+0000 7f30417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:22.377 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.376+0000 7f30417fa640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:34:22.377 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.376+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3044141f10 con 0x7f30440b4d60 2024-03-25T18:34:22.380 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.378+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163725357 0 0) 0x7f3044141c30 con 0x7f30440b1730 2024-03-25T18:34:22.380 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.378+0000 7f30417fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:22.380 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.378+0000 7f30417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:22.380 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.378+0000 7f30417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:22.380 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.378+0000 7f30417fa640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:34:22.380 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.378+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f304413bd10 con 0x7f30440b1730 2024-03-25T18:34:22.387 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.386+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 534988698 0 0) 0x7f3028004eb0 con 0x7f304414be90 2024-03-25T18:34:22.387 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.386+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3044141c30 con 0x7f304414be90 2024-03-25T18:34:22.387 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.386+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f303c001c70 con 0x7f304414be90 2024-03-25T18:34:22.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.386+0000 7f30417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:22.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.386+0000 7f30417fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:22.388 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.386+0000 7f30417fa640 10 monclient(hunting): dump: 2024-03-25T18:34:22.388 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:22.388 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:22.388 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:22.388 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:22.388 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:22.388 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:22.388 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:22.388 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:22.388 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:22.388 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:22.389 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.386+0000 7f30417fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:22.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.394+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2492711968 0 0) 0x7f304413bd10 con 0x7f30440b1730 2024-03-25T18:34:22.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.394+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3028004eb0 con 0x7f30440b1730 2024-03-25T18:34:22.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.394+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f3038001870 con 0x7f30440b1730 2024-03-25T18:34:22.395 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.394+0000 7f30417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:22.396 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.394+0000 7f30417fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:22.396 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.394+0000 7f30417fa640 10 monclient(hunting): dump: 2024-03-25T18:34:22.396 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:22.396 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:22.396 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:22.396 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:22.396 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:22.396 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:22.396 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:22.396 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:22.396 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:22.396 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:22.397 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.396+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 997802038 0 0) 0x7f3044141f10 con 0x7f30440b4d60 2024-03-25T18:34:22.397 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.396+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f304413bd10 con 0x7f30440b4d60 2024-03-25T18:34:22.397 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.396+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f3030001b00 con 0x7f30440b4d60 2024-03-25T18:34:22.397 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.396+0000 7f30417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:22.397 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.396+0000 7f30417fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:22.397 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.396+0000 7f30417fa640 10 monclient(hunting): dump: 2024-03-25T18:34:22.397 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:22.398 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:22.398 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:22.398 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:22.398 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:22.398 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:22.398 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:22.398 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:22.398 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:22.398 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:22.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2835136104 0 0) 0x7f3044141c30 con 0x7f304414be90 2024-03-25T18:34:22.399 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 >> v1:172.21.15.190:6789/0 conn(0x7f30440b1730 legacy=0x7f30441341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 >> v1:172.21.15.99:6789/0 conn(0x7f30440b4d60 legacy=0x7f3044142330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f30417fa640 10 monclient: _finish_hunting 0 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f30417fa640 1 monclient: found mon.c 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f30417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f30417fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3044140810 con 0x7f304414be90 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f30417fa640 10 monclient: _finish_auth 0 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f30417fa640 10 monclient: _check_auth_tickets 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f30417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f304aa93640 5 monclient: authenticate success, global_id 4628 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f304aa93640 10 monclient: _renew_subs 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f304aa93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f304aa93640 1 -- 172.21.15.99:0/4254030341 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3044140d30 con 0x7f304414be90 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f304aa93640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:22.400 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f304aa93640 10 monclient: _renew_subs 2024-03-25T18:34:22.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f304aa93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:22.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f304aa93640 1 -- 172.21.15.99:0/4254030341 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3044140560 con 0x7f304414be90 2024-03-25T18:34:22.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.397+0000 7f304aa93640 1 librados: init done 2024-03-25T18:34:22.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.398+0000 7f304aa93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:22.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.398+0000 7f304aa93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:22.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.398+0000 7f304aa93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:22.401 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.398+0000 7f304aa93640 1 -- 172.21.15.99:0/4254030341 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3044140f20 con 0x7f304414be90 2024-03-25T18:34:22.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.706+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f303c003990 con 0x7f304414be90 2024-03-25T18:34:22.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.706+0000 7f30417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:22.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.706+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f303c003d60 con 0x7f304414be90 2024-03-25T18:34:22.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.706+0000 7f30417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:22.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.706+0000 7f30417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:22.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.706+0000 7f30417fa640 10 monclient: dump: 2024-03-25T18:34:22.707 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:22.707 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:22.707 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:22.708 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:22.708 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:22.708 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:22.708 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:22.708 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:22.708 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:22.708 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:22.708 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.706+0000 7f3042ffd640 4 set_mon_vals no callback set 2024-03-25T18:34:22.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.706+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f303c003f80 con 0x7f304414be90 2024-03-25T18:34:22.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.707+0000 7f304882f640 10 monclient: get_auth_request con 0x7f302806e0f0 auth_method 0 2024-03-25T18:34:22.709 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.707+0000 7f30417fa640 10 client.4628.objecter ms_handle_connect 0x7f302806e0f0 2024-03-25T18:34:22.716 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.714+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f303c0549f0 con 0x7f304414be90 2024-03-25T18:34:22.716 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.714+0000 7f30417fa640 10 client.4628.objecter ms_dispatch 0x7f3044000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:22.716 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.714+0000 7f30417fa640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:22.716 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.714+0000 7f30417fa640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:22.716 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.715+0000 7f30417fa640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-25T18:34:22.716 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.715+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f303c050500 con 0x7f304414be90 2024-03-25T18:34:22.716 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.715+0000 7f30417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:22.716 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.715+0000 7f30417fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:22.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.954+0000 7f304aa93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:22.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.954+0000 7f304aa93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:22.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.954+0000 7f304aa93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:22.956 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:22.954+0000 7f304aa93640 1 -- 172.21.15.99:0/4254030341 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30440627b0 con 0x7f304414be90 2024-03-25T18:34:23.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.095+0000 7f30417fa640 1 -- 172.21.15.99:0/4254030341 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f303c052430 con 0x7f304414be90 2024-03-25T18:34:23.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.095+0000 7f30417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:23.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.095+0000 7f30417fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:23.096 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:34:23.097 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:23.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.097+0000 7f304aa93640 10 librados: watch_flush enter 2024-03-25T18:34:23.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.097+0000 7f304aa93640 10 librados: watch_flush exit 2024-03-25T18:34:23.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.097+0000 7f304aa93640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-25T18:34:23.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.097+0000 7f304aa93640 10 client.4628.objecter successfully canceled tick 2024-03-25T18:34:23.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.097+0000 7f304aa93640 1 -- 172.21.15.99:0/4254030341 >> v1:172.21.15.190:6800/4220409699 conn(0x7f302806e0f0 legacy=0x7f30280705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.097+0000 7f304aa93640 10 monclient: shutdown 2024-03-25T18:34:23.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.097+0000 7f304aa93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:23.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.097+0000 7f304aa93640 1 -- 172.21.15.99:0/4254030341 >> v1:172.21.15.99:6790/0 conn(0x7f304414be90 legacy=0x7f30441e35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.099 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.097+0000 7f304aa93640 1 -- 172.21.15.99:0/4254030341 shutdown_connections 2024-03-25T18:34:23.100 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.097+0000 7f304aa93640 1 -- 172.21.15.99:0/4254030341 >> 172.21.15.99:0/4254030341 conn(0x7f30440617a0 msgr2=0x7f30440b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:23.100 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.097+0000 7f304aa93640 1 -- 172.21.15.99:0/4254030341 shutdown_connections 2024-03-25T18:34:23.100 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.097+0000 7f304aa93640 1 -- 172.21.15.99:0/4254030341 wait complete. 2024-03-25T18:34:23.100 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:23.097+0000 7f304aa93640 1 librados: shutdown 2024-03-25T18:34:23.111 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:29.918 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:30.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.026+0000 7f519a00e640 10 monclient: get_monmap_and_config 2024-03-25T18:34:30.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.026+0000 7f519a00e640 10 monclient: build_initial_monmap 2024-03-25T18:34:30.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.026+0000 7f519a00e640 10 monclient: monmap: 2024-03-25T18:34:30.028 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:34:30.028 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:30.028 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:34:30.027785+0000 2024-03-25T18:34:30.028 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:34:30.027785+0000 2024-03-25T18:34:30.028 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:30.028 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:30.028 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:34:30.028 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:34:30.028 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:34:30.029 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:30.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.027+0000 7f519a00e640 1 Processor -- start 2024-03-25T18:34:30.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.027+0000 7f519a00e640 1 -- start start 2024-03-25T18:34:30.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.027+0000 7f519a00e640 10 monclient: init 2024-03-25T18:34:30.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.028+0000 7f519a00e640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:30.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.028+0000 7f519a00e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:30.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.028+0000 7f519a00e640 10 monclient(hunting): picked mon.noname-b con 0x7f51940a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:34:30.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.028+0000 7f519a00e640 10 monclient(hunting): picked mon.noname-c con 0x7f519413f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:34:30.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.028+0000 7f519a00e640 10 monclient(hunting): picked mon.noname-a con 0x7f51941430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:34:30.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.028+0000 7f519a00e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5194142520 con 0x7f51941430c0 2024-03-25T18:34:30.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.028+0000 7f519a00e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51940b28a0 con 0x7f519413f710 2024-03-25T18:34:30.030 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.028+0000 7f519a00e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51940b2a30 con 0x7f51940a98f0 2024-03-25T18:34:30.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.028+0000 7f519a00e640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:30.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.028+0000 7f519a00e640 10 monclient(hunting): authenticate will time out at 7888.196348s 2024-03-25T18:34:30.034 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.033+0000 7f5192ffd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f519413f710 0x7f51941429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:60156/0 (socket says 172.21.15.99:60156) 2024-03-25T18:34:30.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.034+0000 7f5192ffd640 1 -- 172.21.15.99:0/2724735089 learned_addr learned my addr 172.21.15.99:0/2724735089 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:34:30.052 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.051+0000 7f51927fc640 1 -- 172.21.15.99:0/2724735089 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025062474 0 0) 0x7f51940b28a0 con 0x7f519413f710 2024-03-25T18:34:30.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.051+0000 7f51927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:30.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.051+0000 7f51927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:30.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.051+0000 7f51927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:30.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.051+0000 7f51927fc640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:34:30.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.051+0000 7f51927fc640 1 -- 172.21.15.99:0/2724735089 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f517c003520 con 0x7f519413f710 2024-03-25T18:34:30.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.075+0000 7f51927fc640 1 -- 172.21.15.99:0/2724735089 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380879749 0 0) 0x7f5194142520 con 0x7f51941430c0 2024-03-25T18:34:30.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.075+0000 7f51927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:30.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.075+0000 7f51927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:30.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.075+0000 7f51927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:30.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.075+0000 7f51927fc640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:34:30.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.075+0000 7f51927fc640 1 -- 172.21.15.99:0/2724735089 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51940b28a0 con 0x7f51941430c0 2024-03-25T18:34:30.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.081+0000 7f51927fc640 1 -- 172.21.15.99:0/2724735089 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162606398 0 0) 0x7f51940b2a30 con 0x7f51940a98f0 2024-03-25T18:34:30.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.081+0000 7f51927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:30.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.081+0000 7f51927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:30.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.081+0000 7f51927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:30.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.081+0000 7f51927fc640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:34:30.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.081+0000 7f51927fc640 1 -- 172.21.15.99:0/2724735089 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5194142520 con 0x7f51940a98f0 2024-03-25T18:34:30.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 1 -- 172.21.15.99:0/2724735089 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2985993942 0 0) 0x7f517c003520 con 0x7f519413f710 2024-03-25T18:34:30.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 1 -- 172.21.15.99:0/2724735089 >> v1:172.21.15.99:6790/0 conn(0x7f51940a98f0 legacy=0x7f51940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:30.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 1 -- 172.21.15.99:0/2724735089 >> v1:172.21.15.99:6789/0 conn(0x7f51941430c0 legacy=0x7f519414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:30.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:30.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 1 monclient: found mon.noname-c 2024-03-25T18:34:30.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:30.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:34:30.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 1 -- 172.21.15.99:0/2724735089 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51940b44c0 con 0x7f519413f710 2024-03-25T18:34:30.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:30.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:30.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:30.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 1 -- 172.21.15.99:0/2724735089 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5188001690 con 0x7f519413f710 2024-03-25T18:34:30.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:30.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:30.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f51927fc640 10 monclient: dump: 2024-03-25T18:34:30.088 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:30.088 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:30.088 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:30.088 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:30.088 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:30.088 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:30.088 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:30.088 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:30.088 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:30.088 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:30.088 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f519a00e640 5 monclient: authenticate success, global_id 4636 2024-03-25T18:34:30.089 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.085+0000 7f519a00e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:30.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f51927fc640 1 -- 172.21.15.99:0/2724735089 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51880017f0 con 0x7f519413f710 2024-03-25T18:34:30.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f51927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:30.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f51927fc640 1 -- 172.21.15.99:0/2724735089 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5188001c00 con 0x7f519413f710 2024-03-25T18:34:30.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f51927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:30.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f51927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:30.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f51927fc640 10 monclient: dump: 2024-03-25T18:34:30.109 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:30.109 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:30.109 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:30.109 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:30.109 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:30.109 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:30.109 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:30.109 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:30.110 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:30.110 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:30.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f519a00e640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:30.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f519a00e640 4 set_mon_vals no callback set 2024-03-25T18:34:30.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f519a00e640 10 monclient: shutdown 2024-03-25T18:34:30.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f519a00e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:30.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f519a00e640 1 -- 172.21.15.99:0/2724735089 >> v1:172.21.15.190:6789/0 conn(0x7f519413f710 legacy=0x7f51941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:30.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f519a00e640 1 -- 172.21.15.99:0/2724735089 shutdown_connections 2024-03-25T18:34:30.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f519a00e640 1 -- 172.21.15.99:0/2724735089 >> 172.21.15.99:0/2724735089 conn(0x7f51940b7ea0 msgr2=0x7f51940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:30.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f519a00e640 1 -- 172.21.15.99:0/2724735089 shutdown_connections 2024-03-25T18:34:30.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.108+0000 7f519a00e640 1 -- 172.21.15.99:0/2724735089 wait complete. 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.109+0000 7f519a00e640 10 monclient: build_initial_monmap 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.109+0000 7f519a00e640 10 monclient: monmap: 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.109+0000 7f519a00e640 1 librados: starting msgr at 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.109+0000 7f519a00e640 1 librados: starting objecter 2024-03-25T18:34:30.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.110+0000 7f519a00e640 1 Processor -- start 2024-03-25T18:34:30.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.110+0000 7f519a00e640 1 -- start start 2024-03-25T18:34:30.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.110+0000 7f519a00e640 1 librados: setting wanted keys 2024-03-25T18:34:30.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.110+0000 7f519a00e640 1 librados: calling monclient init 2024-03-25T18:34:30.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.110+0000 7f519a00e640 10 monclient: init 2024-03-25T18:34:30.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.110+0000 7f519a00e640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:30.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.110+0000 7f519a00e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:30.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.110+0000 7f519a00e640 10 monclient(hunting): picked mon.noname-c con 0x7f51940a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:34:30.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.110+0000 7f519a00e640 10 monclient(hunting): picked mon.noname-b con 0x7f519413f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:34:30.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.111+0000 7f519a00e640 10 monclient(hunting): picked mon.noname-a con 0x7f51941430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:34:30.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.111+0000 7f519a00e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51941d77a0 con 0x7f51941430c0 2024-03-25T18:34:30.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.111+0000 7f519a00e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5194138e40 con 0x7f519413f710 2024-03-25T18:34:30.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.111+0000 7f519a00e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51940aba30 con 0x7f51940a98f0 2024-03-25T18:34:30.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.111+0000 7f519a00e640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:30.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.111+0000 7f519a00e640 10 monclient(hunting): authenticate will time out at 7888.279030s 2024-03-25T18:34:30.126 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.125+0000 7f5192ffd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f519413f710 0x7f51941d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:60204/0 (socket says 172.21.15.99:60204) 2024-03-25T18:34:30.126 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.125+0000 7f5192ffd640 1 -- 172.21.15.99:0/3335346132 learned_addr learned my addr 172.21.15.99:0/3335346132 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:34:30.131 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.130+0000 7f517bfff640 10 client.?.objecter ms_handle_connect 0x7f51941430c0 2024-03-25T18:34:30.131 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.130+0000 7f517bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:30.133 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.132+0000 7f517bfff640 10 client.?.objecter ms_handle_connect 0x7f519413f710 2024-03-25T18:34:30.133 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.132+0000 7f517bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:30.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.137+0000 7f517bfff640 10 client.?.objecter ms_handle_connect 0x7f51940a98f0 2024-03-25T18:34:30.138 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.137+0000 7f517bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:30.142 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.141+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485264103 0 0) 0x7f5194138e40 con 0x7f519413f710 2024-03-25T18:34:30.142 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.141+0000 7f517bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:30.142 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.141+0000 7f517bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:30.142 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.141+0000 7f517bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:30.142 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.141+0000 7f517bfff640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:34:30.142 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.141+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5180004eb0 con 0x7f519413f710 2024-03-25T18:34:30.148 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.147+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072262416 0 0) 0x7f51940aba30 con 0x7f51940a98f0 2024-03-25T18:34:30.148 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.147+0000 7f517bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:30.148 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.147+0000 7f517bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:30.148 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.147+0000 7f517bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:30.148 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.147+0000 7f517bfff640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:34:30.148 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.147+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5194138e40 con 0x7f51940a98f0 2024-03-25T18:34:30.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.149+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 705650113 0 0) 0x7f5180004eb0 con 0x7f519413f710 2024-03-25T18:34:30.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.149+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51940aba30 con 0x7f519413f710 2024-03-25T18:34:30.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.149+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5188001bc0 con 0x7f519413f710 2024-03-25T18:34:30.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.149+0000 7f517bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:30.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.149+0000 7f517bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:30.150 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.149+0000 7f517bfff640 10 monclient(hunting): dump: 2024-03-25T18:34:30.150 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:30.151 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:30.151 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:30.151 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:30.151 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:30.151 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:30.151 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:30.151 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:30.151 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:30.151 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:30.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.149+0000 7f517bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:30.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.150+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439630635 0 0) 0x7f51941d77a0 con 0x7f51941430c0 2024-03-25T18:34:30.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.150+0000 7f517bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:30.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.150+0000 7f517bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:30.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.150+0000 7f517bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:30.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.150+0000 7f517bfff640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:34:30.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.150+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5180004eb0 con 0x7f51941430c0 2024-03-25T18:34:30.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.175+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1115077738 0 0) 0x7f5180004eb0 con 0x7f51941430c0 2024-03-25T18:34:30.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.175+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51941d77a0 con 0x7f51941430c0 2024-03-25T18:34:30.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.175+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f518c001bd0 con 0x7f51941430c0 2024-03-25T18:34:30.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.175+0000 7f517bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:30.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.175+0000 7f517bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:30.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.175+0000 7f517bfff640 10 monclient(hunting): dump: 2024-03-25T18:34:30.176 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:30.176 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:30.176 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:30.176 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:30.177 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:30.177 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:30.177 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:30.177 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:30.177 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:30.177 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:30.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.177+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3812488057 0 0) 0x7f5194138e40 con 0x7f51940a98f0 2024-03-25T18:34:30.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.177+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5180004eb0 con 0x7f51940a98f0 2024-03-25T18:34:30.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.177+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5184001870 con 0x7f51940a98f0 2024-03-25T18:34:30.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.177+0000 7f517bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:30.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.177+0000 7f517bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:30.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.177+0000 7f517bfff640 10 monclient(hunting): dump: 2024-03-25T18:34:30.178 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:30.178 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:30.179 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:30.179 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:30.179 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:30.179 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:30.179 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:30.179 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:30.179 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:30.179 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:30.181 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1964768777 0 0) 0x7f51940aba30 con 0x7f519413f710 2024-03-25T18:34:30.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 >> v1:172.21.15.99:6790/0 conn(0x7f51940a98f0 legacy=0x7f5194138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:30.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 >> v1:172.21.15.99:6789/0 conn(0x7f51941430c0 legacy=0x7f5194134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:30.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f517bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:34:30.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f517bfff640 1 monclient: found mon.b 2024-03-25T18:34:30.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f517bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:30.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f517bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:34:30.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51941d8340 con 0x7f519413f710 2024-03-25T18:34:30.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f517bfff640 10 monclient: _finish_auth 0 2024-03-25T18:34:30.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f517bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:34:30.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f517bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:30.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f519a00e640 5 monclient: authenticate success, global_id 4639 2024-03-25T18:34:30.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f519a00e640 10 monclient: _renew_subs 2024-03-25T18:34:30.182 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f519a00e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:34:30.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f519a00e640 1 -- 172.21.15.99:0/3335346132 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5194134820 con 0x7f519413f710 2024-03-25T18:34:30.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f519a00e640 10 client.4639.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:30.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f519a00e640 10 monclient: _renew_subs 2024-03-25T18:34:30.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f519a00e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:34:30.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f519a00e640 1 -- 172.21.15.99:0/3335346132 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51941380c0 con 0x7f519413f710 2024-03-25T18:34:30.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.180+0000 7f519a00e640 1 librados: init done 2024-03-25T18:34:30.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.181+0000 7f519a00e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:30.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.181+0000 7f519a00e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:30.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.181+0000 7f519a00e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:34:30.184 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.181+0000 7f519a00e640 1 -- 172.21.15.99:0/3335346132 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51940627b0 con 0x7f519413f710 2024-03-25T18:34:30.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.493+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5188004a90 con 0x7f519413f710 2024-03-25T18:34:30.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.493+0000 7f517bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:30.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.493+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5188004da0 con 0x7f519413f710 2024-03-25T18:34:30.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.493+0000 7f517bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:30.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.493+0000 7f517bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:30.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.493+0000 7f517bfff640 10 monclient: dump: 2024-03-25T18:34:30.495 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:30.495 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:30.495 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:30.495 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:30.495 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:30.496 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:30.496 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:30.496 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:30.496 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:30.496 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:30.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.493+0000 7f5191ffb640 4 set_mon_vals no callback set 2024-03-25T18:34:30.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.494+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f518801ad70 con 0x7f519413f710 2024-03-25T18:34:30.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.495+0000 7f51937fe640 10 monclient: get_auth_request con 0x7f518006e110 auth_method 0 2024-03-25T18:34:30.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.495+0000 7f517bfff640 10 client.4639.objecter ms_handle_connect 0x7f518006e110 2024-03-25T18:34:30.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.500+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f5188054910 con 0x7f519413f710 2024-03-25T18:34:30.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.500+0000 7f517bfff640 10 client.4639.objecter ms_dispatch 0x7f5194000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:30.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.500+0000 7f517bfff640 3 client.4639.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:30.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.500+0000 7f517bfff640 3 client.4639.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:30.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.500+0000 7f517bfff640 20 client.4639.objecter dump_active .. 0 homeless 2024-03-25T18:34:30.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.500+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5188050420 con 0x7f519413f710 2024-03-25T18:34:30.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.500+0000 7f517bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:30.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.500+0000 7f517bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:30.749 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.748+0000 7f519a00e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:30.749 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.748+0000 7f519a00e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:30.749 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.748+0000 7f519a00e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:34:30.749 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.748+0000 7f519a00e640 1 -- 172.21.15.99:0/3335346132 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f519414e880 con 0x7f519413f710 2024-03-25T18:34:30.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.774+0000 7f517bfff640 1 -- 172.21.15.99:0/3335346132 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f5188052350 con 0x7f519413f710 2024-03-25T18:34:30.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.774+0000 7f517bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:30.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.774+0000 7f517bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:30.776 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:34:30.776 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:30.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.776+0000 7f519a00e640 10 librados: watch_flush enter 2024-03-25T18:34:30.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.777+0000 7f519a00e640 10 librados: watch_flush exit 2024-03-25T18:34:30.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.777+0000 7f519a00e640 20 client.4639.objecter shutdown clearing up homeless session... 2024-03-25T18:34:30.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.777+0000 7f519a00e640 10 client.4639.objecter successfully canceled tick 2024-03-25T18:34:30.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.777+0000 7f519a00e640 1 -- 172.21.15.99:0/3335346132 >> v1:172.21.15.190:6800/4220409699 conn(0x7f518006e110 legacy=0x7f51800705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:30.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.777+0000 7f519a00e640 10 monclient: shutdown 2024-03-25T18:34:30.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.777+0000 7f519a00e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:30.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.777+0000 7f519a00e640 1 -- 172.21.15.99:0/3335346132 >> v1:172.21.15.190:6789/0 conn(0x7f519413f710 legacy=0x7f51941d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:30.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.777+0000 7f519a00e640 1 -- 172.21.15.99:0/3335346132 shutdown_connections 2024-03-25T18:34:30.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.777+0000 7f519a00e640 1 -- 172.21.15.99:0/3335346132 >> 172.21.15.99:0/3335346132 conn(0x7f51940b7ea0 msgr2=0x7f5194074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:30.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.777+0000 7f519a00e640 1 -- 172.21.15.99:0/3335346132 shutdown_connections 2024-03-25T18:34:30.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.777+0000 7f519a00e640 1 -- 172.21.15.99:0/3335346132 wait complete. 2024-03-25T18:34:30.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:30.777+0000 7f519a00e640 1 librados: shutdown 2024-03-25T18:34:30.790 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:35.101 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:09:35.694 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:35.107 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:09:35.699 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:35.145 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:09:35.737 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:35.152 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:09:35.711 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:35.166 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:09:35.759 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:35.171 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:09:35.730 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:35.180 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:09:35.773 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:35.194 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:09:35.752 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:35.200 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:09:35.793 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:37.597 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:37.714 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.713+0000 7effc504e640 10 monclient: get_monmap_and_config 2024-03-25T18:34:37.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.713+0000 7effc504e640 10 monclient: build_initial_monmap 2024-03-25T18:34:37.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.713+0000 7effc504e640 10 monclient: monmap: 2024-03-25T18:34:37.715 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:34:37.715 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:37.715 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:34:37.714820+0000 2024-03-25T18:34:37.715 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:34:37.714820+0000 2024-03-25T18:34:37.715 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:37.715 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:37.715 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:34:37.715 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:34:37.716 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:34:37.716 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:37.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.714+0000 7effc504e640 1 Processor -- start 2024-03-25T18:34:37.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.714+0000 7effc504e640 1 -- start start 2024-03-25T18:34:37.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.714+0000 7effc504e640 10 monclient: init 2024-03-25T18:34:37.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.715+0000 7effc504e640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:37.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.715+0000 7effc504e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:37.717 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.715+0000 7effc504e640 10 monclient(hunting): picked mon.noname-c con 0x7effc00b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:34:37.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.715+0000 7effc504e640 10 monclient(hunting): picked mon.noname-b con 0x7effc00b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:34:37.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.715+0000 7effc504e640 10 monclient(hunting): picked mon.noname-a con 0x7effc014be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:34:37.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.715+0000 7effc504e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc00af940 con 0x7effc014be90 2024-03-25T18:34:37.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.715+0000 7effc504e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc00b0a70 con 0x7effc00b4d60 2024-03-25T18:34:37.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.715+0000 7effc504e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc00b0d50 con 0x7effc00b1730 2024-03-25T18:34:37.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.715+0000 7effc504e640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:37.718 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.715+0000 7effc504e640 10 monclient(hunting): authenticate will time out at 7895.883014s 2024-03-25T18:34:37.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.725+0000 7effbed9d640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7effc00b4d60 0x7effc013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:59436/0 (socket says 172.21.15.99:59436) 2024-03-25T18:34:37.727 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.725+0000 7effbed9d640 1 -- 172.21.15.99:0/2489766791 learned_addr learned my addr 172.21.15.99:0/2489766791 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:34:37.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.738+0000 7effbdd9b640 1 -- 172.21.15.99:0/2489766791 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655525943 0 0) 0x7effc00b0a70 con 0x7effc00b4d60 2024-03-25T18:34:37.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.738+0000 7effbdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:37.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.738+0000 7effbdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:37.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.738+0000 7effbdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:37.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.738+0000 7effbdd9b640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:34:37.739 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.738+0000 7effbdd9b640 1 -- 172.21.15.99:0/2489766791 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effa8003520 con 0x7effc00b4d60 2024-03-25T18:34:37.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.760+0000 7effbdd9b640 1 -- 172.21.15.99:0/2489766791 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075613775 0 0) 0x7effc00af940 con 0x7effc014be90 2024-03-25T18:34:37.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.760+0000 7effbdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:34:37.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.760+0000 7effbdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:37.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.760+0000 7effbdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:37.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.760+0000 7effbdd9b640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:34:37.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.760+0000 7effbdd9b640 1 -- 172.21.15.99:0/2489766791 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc00b0a70 con 0x7effc014be90 2024-03-25T18:34:37.838 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 1 -- 172.21.15.99:0/2489766791 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1291593148 0 0) 0x7effc00b0a70 con 0x7effc014be90 2024-03-25T18:34:37.838 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 1 -- 172.21.15.99:0/2489766791 >> v1:172.21.15.99:6790/0 conn(0x7effc00b1730 legacy=0x7effc00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:37.838 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 1 -- 172.21.15.99:0/2489766791 >> v1:172.21.15.190:6789/0 conn(0x7effc00b4d60 legacy=0x7effc013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 1 monclient: found mon.noname-a 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 1 -- 172.21.15.99:0/2489766791 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc00b1030 con 0x7effc014be90 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 10 monclient: _finish_auth 0 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 1 -- 172.21.15.99:0/2489766791 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7effb80019f0 con 0x7effc014be90 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effbdd9b640 10 monclient: dump: 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:37.839 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:37.840 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:37.840 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:37.840 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:37.840 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:37.840 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:37.840 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:37.840 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:37.840 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effc504e640 5 monclient: authenticate success, global_id 4632 2024-03-25T18:34:37.840 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.837+0000 7effc504e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:37.855 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.853+0000 7effbdd9b640 1 -- 172.21.15.99:0/2489766791 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effb8001b50 con 0x7effc014be90 2024-03-25T18:34:37.855 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.853+0000 7effbdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:37.855 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.853+0000 7effbdd9b640 1 -- 172.21.15.99:0/2489766791 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7effb8001f20 con 0x7effc014be90 2024-03-25T18:34:37.855 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.853+0000 7effbdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:37.855 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.853+0000 7effbdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:37.855 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.853+0000 7effbdd9b640 10 monclient: dump: 2024-03-25T18:34:37.855 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:37.855 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:37.855 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:37.855 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:37.855 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:37.855 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:37.855 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:37.855 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:37.856 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:37.856 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:37.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.853+0000 7effc504e640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:37.856 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.853+0000 7effc504e640 4 set_mon_vals no callback set 2024-03-25T18:34:37.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.854+0000 7effc504e640 10 monclient: shutdown 2024-03-25T18:34:37.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.854+0000 7effc504e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:37.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.854+0000 7effc504e640 1 -- 172.21.15.99:0/2489766791 >> v1:172.21.15.99:6789/0 conn(0x7effc014be90 legacy=0x7effc014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:37.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.854+0000 7effc504e640 1 -- 172.21.15.99:0/2489766791 shutdown_connections 2024-03-25T18:34:37.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.854+0000 7effc504e640 1 -- 172.21.15.99:0/2489766791 >> 172.21.15.99:0/2489766791 conn(0x7effc00617a0 msgr2=0x7effc0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:37.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.854+0000 7effc504e640 1 -- 172.21.15.99:0/2489766791 shutdown_connections 2024-03-25T18:34:37.857 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.854+0000 7effc504e640 1 -- 172.21.15.99:0/2489766791 wait complete. 2024-03-25T18:34:37.858 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.855+0000 7effc504e640 10 monclient: build_initial_monmap 2024-03-25T18:34:37.858 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.855+0000 7effc504e640 10 monclient: monmap: 2024-03-25T18:34:37.858 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:34:37.858 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:37.858 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:34:37.858 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:34:37.858 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:37.858 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:34:37.858 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:34:37.858 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:34:37.858 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:34:37.858 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:37.858 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.856+0000 7effc504e640 1 librados: starting msgr at 2024-03-25T18:34:37.858 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.856+0000 7effc504e640 1 librados: starting objecter 2024-03-25T18:34:37.859 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.856+0000 7effc504e640 1 Processor -- start 2024-03-25T18:34:37.859 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.856+0000 7effc504e640 1 -- start start 2024-03-25T18:34:37.859 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.856+0000 7effc504e640 1 librados: setting wanted keys 2024-03-25T18:34:37.859 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.856+0000 7effc504e640 1 librados: calling monclient init 2024-03-25T18:34:37.859 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.856+0000 7effc504e640 10 monclient: init 2024-03-25T18:34:37.859 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.856+0000 7effc504e640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:37.859 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.856+0000 7effc504e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:37.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.856+0000 7effc504e640 10 monclient(hunting): picked mon.noname-c con 0x7effc00b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:34:37.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.857+0000 7effc504e640 10 monclient(hunting): picked mon.noname-b con 0x7effc00b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:34:37.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.857+0000 7effc504e640 10 monclient(hunting): picked mon.noname-a con 0x7effc014be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:34:37.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.857+0000 7effc504e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc01d8360 con 0x7effc014be90 2024-03-25T18:34:37.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.857+0000 7effc504e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc013b0f0 con 0x7effc00b4d60 2024-03-25T18:34:37.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.857+0000 7effc504e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc0138720 con 0x7effc00b1730 2024-03-25T18:34:37.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.857+0000 7effc504e640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:37.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.857+0000 7effc504e640 10 monclient(hunting): authenticate will time out at 7896.024985s 2024-03-25T18:34:37.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.862+0000 7effbf59e640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7effc014be90 0x7effc0138010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:60106/0 (socket says 172.21.15.99:60106) 2024-03-25T18:34:37.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.862+0000 7effbf59e640 1 -- 172.21.15.99:0/79957234 learned_addr learned my addr 172.21.15.99:0/79957234 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:34:37.864 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.863+0000 7effa77fe640 10 client.?.objecter ms_handle_connect 0x7effc014be90 2024-03-25T18:34:37.864 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.863+0000 7effa77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:37.869 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.868+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262196466 0 0) 0x7effc01d8360 con 0x7effc014be90 2024-03-25T18:34:37.869 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.868+0000 7effa77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:37.869 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.868+0000 7effa77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:37.869 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.868+0000 7effa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:37.869 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.868+0000 7effa77fe640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:34:37.869 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.868+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effac004eb0 con 0x7effc014be90 2024-03-25T18:34:37.889 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.887+0000 7effa77fe640 10 client.?.objecter ms_handle_connect 0x7effc00b4d60 2024-03-25T18:34:37.889 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.887+0000 7effa77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:37.889 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.887+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1849824421 0 0) 0x7effac004eb0 con 0x7effc014be90 2024-03-25T18:34:37.889 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.887+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effc01d8360 con 0x7effc014be90 2024-03-25T18:34:37.889 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.887+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7effb80019b0 con 0x7effc014be90 2024-03-25T18:34:37.889 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.887+0000 7effa77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:37.889 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.887+0000 7effa77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:37.889 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.887+0000 7effa77fe640 10 monclient(hunting): dump: 2024-03-25T18:34:37.889 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:37.889 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:37.889 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:37.889 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:37.889 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:37.890 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:37.890 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:37.890 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:37.890 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:37.890 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:37.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.887+0000 7effa77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:37.893 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.892+0000 7effa77fe640 10 client.?.objecter ms_handle_connect 0x7effc00b1730 2024-03-25T18:34:37.893 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.892+0000 7effa77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:37.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.896+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833462962 0 0) 0x7effc0138720 con 0x7effc00b1730 2024-03-25T18:34:37.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.896+0000 7effa77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:37.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.896+0000 7effa77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:37.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.896+0000 7effa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:37.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.896+0000 7effa77fe640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:34:37.897 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.896+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effac004eb0 con 0x7effc00b1730 2024-03-25T18:34:37.901 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.901+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888498139 0 0) 0x7effc013b0f0 con 0x7effc00b4d60 2024-03-25T18:34:37.902 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.901+0000 7effa77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:37.902 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.901+0000 7effa77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:37.902 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.901+0000 7effa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:37.902 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.901+0000 7effa77fe640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:34:37.902 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.901+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc0138720 con 0x7effc00b4d60 2024-03-25T18:34:37.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2701719309 0 0) 0x7effc01d8360 con 0x7effc014be90 2024-03-25T18:34:37.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 >> v1:172.21.15.99:6790/0 conn(0x7effc00b1730 legacy=0x7effc013bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:37.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 >> v1:172.21.15.190:6789/0 conn(0x7effc00b4d60 legacy=0x7effc01d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:37.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effa77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:37.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effa77fe640 1 monclient: found mon.a 2024-03-25T18:34:37.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effa77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:37.907 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effa77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:34:37.907 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc01388b0 con 0x7effc014be90 2024-03-25T18:34:37.907 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effa77fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:37.907 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effa77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:37.907 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effa77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:37.907 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effc504e640 5 monclient: authenticate success, global_id 4635 2024-03-25T18:34:37.907 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effc504e640 10 monclient: _renew_subs 2024-03-25T18:34:37.907 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effc504e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:34:37.907 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effc504e640 1 -- 172.21.15.99:0/79957234 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc013b380 con 0x7effc014be90 2024-03-25T18:34:37.907 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effc504e640 10 client.4635.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:37.907 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effc504e640 10 monclient: _renew_subs 2024-03-25T18:34:37.907 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effc504e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:34:37.907 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effc504e640 1 -- 172.21.15.99:0/79957234 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7effc0138fd0 con 0x7effc014be90 2024-03-25T18:34:37.907 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.905+0000 7effc504e640 1 librados: init done 2024-03-25T18:34:37.908 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.906+0000 7effc504e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:37.908 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.906+0000 7effc504e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:37.908 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.906+0000 7effc504e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:34:37.908 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:37.906+0000 7effc504e640 1 -- 172.21.15.99:0/79957234 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc00aa770 con 0x7effc014be90 2024-03-25T18:34:38.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.205+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effb8003c20 con 0x7effc014be90 2024-03-25T18:34:38.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.205+0000 7effa77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:38.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.205+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7effb8003fb0 con 0x7effc014be90 2024-03-25T18:34:38.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.205+0000 7effa77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:38.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.205+0000 7effa77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:38.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.205+0000 7effa77fe640 10 monclient: dump: 2024-03-25T18:34:38.207 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:38.207 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:38.207 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:38.207 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:38.207 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:38.207 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:38.207 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:38.207 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:38.207 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:38.207 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:38.207 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.205+0000 7effbd59a640 4 set_mon_vals no callback set 2024-03-25T18:34:38.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.205+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7effb80041d0 con 0x7effc014be90 2024-03-25T18:34:38.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.206+0000 7effbed9d640 10 monclient: get_auth_request con 0x7effac06dd70 auth_method 0 2024-03-25T18:34:38.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.206+0000 7effa77fe640 10 client.4635.objecter ms_handle_connect 0x7effac06dd70 2024-03-25T18:34:38.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.212+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7effb8054c00 con 0x7effc014be90 2024-03-25T18:34:38.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.212+0000 7effa77fe640 10 client.4635.objecter ms_dispatch 0x7effc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:38.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.212+0000 7effa77fe640 3 client.4635.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:38.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.212+0000 7effa77fe640 3 client.4635.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:38.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.212+0000 7effa77fe640 20 client.4635.objecter dump_active .. 0 homeless 2024-03-25T18:34:38.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.212+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7effb8050710 con 0x7effc014be90 2024-03-25T18:34:38.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.212+0000 7effa77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:38.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.212+0000 7effa77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:38.470 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.469+0000 7effc504e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:38.470 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.469+0000 7effc504e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:38.470 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.469+0000 7effc504e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:34:38.470 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.469+0000 7effc504e640 1 -- 172.21.15.99:0/79957234 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effc00627b0 con 0x7effc014be90 2024-03-25T18:34:38.494 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.492+0000 7effa77fe640 1 -- 172.21.15.99:0/79957234 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7effb8052640 con 0x7effc014be90 2024-03-25T18:34:38.494 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.493+0000 7effa77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:38.494 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.493+0000 7effa77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:38.494 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:34:38.494 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:38.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.495+0000 7effc504e640 10 librados: watch_flush enter 2024-03-25T18:34:38.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.495+0000 7effc504e640 10 librados: watch_flush exit 2024-03-25T18:34:38.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.495+0000 7effc504e640 20 client.4635.objecter shutdown clearing up homeless session... 2024-03-25T18:34:38.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.495+0000 7effc504e640 10 client.4635.objecter successfully canceled tick 2024-03-25T18:34:38.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.495+0000 7effc504e640 1 -- 172.21.15.99:0/79957234 >> v1:172.21.15.190:6800/4220409699 conn(0x7effac06dd70 legacy=0x7effac070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:38.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.495+0000 7effc504e640 10 monclient: shutdown 2024-03-25T18:34:38.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.495+0000 7effc504e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:38.498 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.495+0000 7effc504e640 1 -- 172.21.15.99:0/79957234 >> v1:172.21.15.99:6789/0 conn(0x7effc014be90 legacy=0x7effc0138010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:38.498 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.496+0000 7effc504e640 1 -- 172.21.15.99:0/79957234 shutdown_connections 2024-03-25T18:34:38.498 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.496+0000 7effc504e640 1 -- 172.21.15.99:0/79957234 >> 172.21.15.99:0/79957234 conn(0x7effc00617a0 msgr2=0x7effc0145150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:38.498 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.496+0000 7effc504e640 1 -- 172.21.15.99:0/79957234 shutdown_connections 2024-03-25T18:34:38.498 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.496+0000 7effc504e640 1 -- 172.21.15.99:0/79957234 wait complete. 2024-03-25T18:34:38.498 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:38.496+0000 7effc504e640 1 librados: shutdown 2024-03-25T18:34:38.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:45.316 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:45.433 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.432+0000 7fb72c19d640 10 monclient: get_monmap_and_config 2024-03-25T18:34:45.434 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.432+0000 7fb72c19d640 10 monclient: build_initial_monmap 2024-03-25T18:34:45.434 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.432+0000 7fb72c19d640 10 monclient: monmap: 2024-03-25T18:34:45.434 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:34:45.434 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:45.434 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:34:45.433873+0000 2024-03-25T18:34:45.434 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:34:45.433873+0000 2024-03-25T18:34:45.434 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:45.434 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:45.434 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:34:45.434 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:34:45.434 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:34:45.434 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:45.436 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.433+0000 7fb72c19d640 1 Processor -- start 2024-03-25T18:34:45.436 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.433+0000 7fb72c19d640 1 -- start start 2024-03-25T18:34:45.436 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.433+0000 7fb72c19d640 10 monclient: init 2024-03-25T18:34:45.436 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.434+0000 7fb72c19d640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:45.436 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.434+0000 7fb72c19d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:45.436 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.434+0000 7fb72c19d640 10 monclient(hunting): picked mon.noname-b con 0x7fb7240a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:34:45.436 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.434+0000 7fb72c19d640 10 monclient(hunting): picked mon.noname-c con 0x7fb72413f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:34:45.436 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.434+0000 7fb72c19d640 10 monclient(hunting): picked mon.noname-a con 0x7fb7241430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:34:45.436 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.434+0000 7fb72c19d640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb724142520 con 0x7fb7241430c0 2024-03-25T18:34:45.436 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.434+0000 7fb72c19d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7240b28a0 con 0x7fb72413f710 2024-03-25T18:34:45.436 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.434+0000 7fb72c19d640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7240b2a30 con 0x7fb7240a98f0 2024-03-25T18:34:45.437 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.434+0000 7fb72c19d640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:45.437 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.434+0000 7fb72c19d640 10 monclient(hunting): authenticate will time out at 7903.602829s 2024-03-25T18:34:45.443 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.442+0000 7fb729738640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb72413f710 0x7fb7241429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:40128/0 (socket says 172.21.15.99:40128) 2024-03-25T18:34:45.443 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.442+0000 7fb729738640 1 -- 172.21.15.99:0/4275851942 learned_addr learned my addr 172.21.15.99:0/4275851942 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:34:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.455+0000 7fb728f37640 1 -- 172.21.15.99:0/4275851942 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312235656 0 0) 0x7fb7240b28a0 con 0x7fb72413f710 2024-03-25T18:34:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.455+0000 7fb728f37640 10 monclient(hunting): _negotiate 2024-03-25T18:34:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.455+0000 7fb728f37640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.455+0000 7fb728f37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.455+0000 7fb728f37640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:34:45.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.455+0000 7fb728f37640 1 -- 172.21.15.99:0/4275851942 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb710003520 con 0x7fb72413f710 2024-03-25T18:34:45.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.460+0000 7fb728f37640 1 -- 172.21.15.99:0/4275851942 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567363609 0 0) 0x7fb7240b2a30 con 0x7fb7240a98f0 2024-03-25T18:34:45.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.460+0000 7fb728f37640 10 monclient(hunting): _negotiate 2024-03-25T18:34:45.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.460+0000 7fb728f37640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:45.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.460+0000 7fb728f37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:45.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.460+0000 7fb728f37640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:34:45.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.460+0000 7fb728f37640 1 -- 172.21.15.99:0/4275851942 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7240b28a0 con 0x7fb7240a98f0 2024-03-25T18:34:45.464 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.463+0000 7fb728f37640 1 -- 172.21.15.99:0/4275851942 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173973495 0 0) 0x7fb724142520 con 0x7fb7241430c0 2024-03-25T18:34:45.464 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.463+0000 7fb728f37640 10 monclient(hunting): _negotiate 2024-03-25T18:34:45.464 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.463+0000 7fb728f37640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:45.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.463+0000 7fb728f37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:45.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.463+0000 7fb728f37640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:34:45.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.463+0000 7fb728f37640 1 -- 172.21.15.99:0/4275851942 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7240b2a30 con 0x7fb7241430c0 2024-03-25T18:34:45.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 1 -- 172.21.15.99:0/4275851942 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4238796020 0 0) 0x7fb710003520 con 0x7fb72413f710 2024-03-25T18:34:45.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 1 -- 172.21.15.99:0/4275851942 >> v1:172.21.15.99:6790/0 conn(0x7fb7240a98f0 legacy=0x7fb7240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:45.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 1 -- 172.21.15.99:0/4275851942 >> v1:172.21.15.99:6789/0 conn(0x7fb7241430c0 legacy=0x7fb72414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:45.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 10 monclient: _finish_hunting 0 2024-03-25T18:34:45.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 1 monclient: found mon.noname-c 2024-03-25T18:34:45.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:45.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:34:45.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 1 -- 172.21.15.99:0/4275851942 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7240b44c0 con 0x7fb72413f710 2024-03-25T18:34:45.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 10 monclient: _finish_auth 0 2024-03-25T18:34:45.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 10 monclient: _check_auth_tickets 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb72c19d640 5 monclient: authenticate success, global_id 4648 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 1 -- 172.21.15.99:0/4275851942 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb714001690 con 0x7fb72413f710 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb728f37640 10 monclient: dump: 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:45.468 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:45.469 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.465+0000 7fb72c19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:45.490 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.489+0000 7fb728f37640 1 -- 172.21.15.99:0/4275851942 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7140017f0 con 0x7fb72413f710 2024-03-25T18:34:45.491 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.489+0000 7fb728f37640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:45.491 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.489+0000 7fb728f37640 1 -- 172.21.15.99:0/4275851942 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb714001c00 con 0x7fb72413f710 2024-03-25T18:34:45.491 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.489+0000 7fb72c19d640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:45.491 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.489+0000 7fb72c19d640 4 set_mon_vals no callback set 2024-03-25T18:34:45.491 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.489+0000 7fb728f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:45.491 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.489+0000 7fb728f37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:45.491 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.489+0000 7fb728f37640 10 monclient: dump: 2024-03-25T18:34:45.491 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:45.491 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:45.491 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:45.491 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:45.491 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:45.491 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:45.491 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:45.492 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:45.492 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:45.492 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:45.493 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.490+0000 7fb72c19d640 10 monclient: shutdown 2024-03-25T18:34:45.493 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.490+0000 7fb72c19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:45.493 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.490+0000 7fb72c19d640 1 -- 172.21.15.99:0/4275851942 >> v1:172.21.15.190:6789/0 conn(0x7fb72413f710 legacy=0x7fb7241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:45.493 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.490+0000 7fb72c19d640 1 -- 172.21.15.99:0/4275851942 shutdown_connections 2024-03-25T18:34:45.494 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.490+0000 7fb72c19d640 1 -- 172.21.15.99:0/4275851942 >> 172.21.15.99:0/4275851942 conn(0x7fb7240b7ea0 msgr2=0x7fb7240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:45.494 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.491+0000 7fb72c19d640 1 -- 172.21.15.99:0/4275851942 shutdown_connections 2024-03-25T18:34:45.494 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.491+0000 7fb72c19d640 1 -- 172.21.15.99:0/4275851942 wait complete. 2024-03-25T18:34:45.494 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.491+0000 7fb72c19d640 10 monclient: build_initial_monmap 2024-03-25T18:34:45.494 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.491+0000 7fb72c19d640 10 monclient: monmap: 2024-03-25T18:34:45.494 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:34:45.494 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:45.494 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:34:45.494 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:34:45.494 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:45.494 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:34:45.494 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:34:45.494 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:34:45.495 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:34:45.495 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:45.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.492+0000 7fb72c19d640 1 librados: starting msgr at 2024-03-25T18:34:45.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.492+0000 7fb72c19d640 1 librados: starting objecter 2024-03-25T18:34:45.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.492+0000 7fb72c19d640 1 Processor -- start 2024-03-25T18:34:45.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.492+0000 7fb72c19d640 1 -- start start 2024-03-25T18:34:45.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.492+0000 7fb72c19d640 1 librados: setting wanted keys 2024-03-25T18:34:45.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.492+0000 7fb72c19d640 1 librados: calling monclient init 2024-03-25T18:34:45.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.492+0000 7fb72c19d640 10 monclient: init 2024-03-25T18:34:45.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.493+0000 7fb72c19d640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:45.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.493+0000 7fb72c19d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:45.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.493+0000 7fb72c19d640 10 monclient(hunting): picked mon.noname-b con 0x7fb7240a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:34:45.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.493+0000 7fb72c19d640 10 monclient(hunting): picked mon.noname-c con 0x7fb72413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:34:45.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.493+0000 7fb72c19d640 10 monclient(hunting): picked mon.noname-a con 0x7fb7241430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:34:45.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.493+0000 7fb72c19d640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7241d7490 con 0x7fb7241430c0 2024-03-25T18:34:45.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.493+0000 7fb72c19d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7241d76c0 con 0x7fb7240a98f0 2024-03-25T18:34:45.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.493+0000 7fb72c19d640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7241d82f0 con 0x7fb72413f710 2024-03-25T18:34:45.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.493+0000 7fb72c19d640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:45.497 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.493+0000 7fb72c19d640 10 monclient(hunting): authenticate will time out at 7903.661242s 2024-03-25T18:34:45.503 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.503+0000 7fb729f39640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb7240a98f0 0x7fb7241d7be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:40170/0 (socket says 172.21.15.99:40170) 2024-03-25T18:34:45.504 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.503+0000 7fb729f39640 1 -- 172.21.15.99:0/4207110946 learned_addr learned my addr 172.21.15.99:0/4207110946 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:34:45.504 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.504+0000 7fb71a7fc640 10 client.?.objecter ms_handle_connect 0x7fb7240a98f0 2024-03-25T18:34:45.505 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.504+0000 7fb71a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:45.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.511+0000 7fb71a7fc640 10 client.?.objecter ms_handle_connect 0x7fb7241430c0 2024-03-25T18:34:45.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.511+0000 7fb71a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:45.513 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.512+0000 7fb71a7fc640 10 client.?.objecter ms_handle_connect 0x7fb72413f710 2024-03-25T18:34:45.513 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.512+0000 7fb71a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:45.516 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.515+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183836935 0 0) 0x7fb7241d76c0 con 0x7fb7240a98f0 2024-03-25T18:34:45.516 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.515+0000 7fb71a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:45.516 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.515+0000 7fb71a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:45.516 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.515+0000 7fb71a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:45.516 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.515+0000 7fb71a7fc640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:34:45.516 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.515+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb708004eb0 con 0x7fb7240a98f0 2024-03-25T18:34:45.537 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.536+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710512555 0 0) 0x7fb7241d7490 con 0x7fb7241430c0 2024-03-25T18:34:45.537 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.536+0000 7fb71a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:45.537 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.536+0000 7fb71a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:45.538 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.536+0000 7fb71a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:45.538 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.536+0000 7fb71a7fc640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:34:45.538 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.536+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7241d76c0 con 0x7fb7241430c0 2024-03-25T18:34:45.541 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.540+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1724428981 0 0) 0x7fb708004eb0 con 0x7fb7240a98f0 2024-03-25T18:34:45.541 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.540+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7241d7490 con 0x7fb7240a98f0 2024-03-25T18:34:45.541 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.540+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb720001a30 con 0x7fb7240a98f0 2024-03-25T18:34:45.541 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.540+0000 7fb71a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:45.541 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.540+0000 7fb71a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:45.541 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.540+0000 7fb71a7fc640 10 monclient(hunting): dump: 2024-03-25T18:34:45.541 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:45.541 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:45.541 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:45.541 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:45.541 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:45.541 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:45.542 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:45.542 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:45.542 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:45.542 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:45.542 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.540+0000 7fb71a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:45.548 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.547+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783925743 0 0) 0x7fb7241d82f0 con 0x7fb72413f710 2024-03-25T18:34:45.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.547+0000 7fb71a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:45.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.547+0000 7fb71a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:45.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.547+0000 7fb71a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:45.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.547+0000 7fb71a7fc640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:34:45.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.547+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb708004eb0 con 0x7fb72413f710 2024-03-25T18:34:45.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.547+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1044456678 0 0) 0x7fb7241d76c0 con 0x7fb7241430c0 2024-03-25T18:34:45.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.547+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7241d82f0 con 0x7fb7241430c0 2024-03-25T18:34:45.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.547+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb71c001b50 con 0x7fb7241430c0 2024-03-25T18:34:45.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.547+0000 7fb71a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:45.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.547+0000 7fb71a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:45.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.547+0000 7fb71a7fc640 10 monclient(hunting): dump: 2024-03-25T18:34:45.549 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:45.549 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:45.549 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:45.550 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:45.550 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:45.550 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:45.550 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:45.550 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:45.550 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:45.550 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:45.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.551+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3055572454 0 0) 0x7fb7241d7490 con 0x7fb7240a98f0 2024-03-25T18:34:45.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 >> v1:172.21.15.99:6790/0 conn(0x7fb72413f710 legacy=0x7fb7241383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:45.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 >> v1:172.21.15.99:6789/0 conn(0x7fb7241430c0 legacy=0x7fb724133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:45.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb71a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:45.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb71a7fc640 1 monclient: found mon.b 2024-03-25T18:34:45.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb71a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:45.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb71a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:34:45.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb724138af0 con 0x7fb7240a98f0 2024-03-25T18:34:45.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb71a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:45.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb71a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:45.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb71a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:45.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb72c19d640 5 monclient: authenticate success, global_id 4651 2024-03-25T18:34:45.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb72c19d640 10 monclient: _renew_subs 2024-03-25T18:34:45.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb72c19d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:34:45.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb72c19d640 1 -- 172.21.15.99:0/4207110946 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7241376d0 con 0x7fb7240a98f0 2024-03-25T18:34:45.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb72c19d640 10 client.4651.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:45.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb72c19d640 10 monclient: _renew_subs 2024-03-25T18:34:45.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb72c19d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:34:45.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb72c19d640 1 -- 172.21.15.99:0/4207110946 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb724134fc0 con 0x7fb7240a98f0 2024-03-25T18:34:45.554 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.552+0000 7fb72c19d640 1 librados: init done 2024-03-25T18:34:45.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.553+0000 7fb72c19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:45.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.553+0000 7fb72c19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:45.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.553+0000 7fb72c19d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:34:45.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.553+0000 7fb72c19d640 1 -- 172.21.15.99:0/4207110946 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7241d7850 con 0x7fb7240a98f0 2024-03-25T18:34:45.827 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.826+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb720001c70 con 0x7fb7240a98f0 2024-03-25T18:34:45.827 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.826+0000 7fb71a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:45.827 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.826+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb720003a60 con 0x7fb7240a98f0 2024-03-25T18:34:45.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.826+0000 7fb71a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:45.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.826+0000 7fb71a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:45.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.826+0000 7fb71a7fc640 10 monclient: dump: 2024-03-25T18:34:45.828 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:45.828 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:45.828 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:45.828 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:45.828 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:45.828 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:45.828 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:45.828 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:45.828 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:45.828 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:45.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.826+0000 7fb71bfff640 4 set_mon_vals no callback set 2024-03-25T18:34:45.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.827+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fb720003c40 con 0x7fb7240a98f0 2024-03-25T18:34:45.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.828+0000 7fb729738640 10 monclient: get_auth_request con 0x7fb70806dfb0 auth_method 0 2024-03-25T18:34:45.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.828+0000 7fb71a7fc640 10 client.4651.objecter ms_handle_connect 0x7fb70806dfb0 2024-03-25T18:34:45.833 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.832+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7fb7200548d0 con 0x7fb7240a98f0 2024-03-25T18:34:45.833 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.832+0000 7fb71a7fc640 10 client.4651.objecter ms_dispatch 0x7fb724000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:45.833 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.832+0000 7fb71a7fc640 3 client.4651.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:45.833 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.832+0000 7fb71a7fc640 3 client.4651.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:45.833 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.832+0000 7fb71a7fc640 20 client.4651.objecter dump_active .. 0 homeless 2024-03-25T18:34:45.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.833+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7200503e0 con 0x7fb7240a98f0 2024-03-25T18:34:45.834 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.833+0000 7fb71a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:45.835 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:45.833+0000 7fb71a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:46.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.085+0000 7fb72c19d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:46.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.085+0000 7fb72c19d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:46.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.085+0000 7fb72c19d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:34:46.087 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.085+0000 7fb72c19d640 1 -- 172.21.15.99:0/4207110946 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb72414f300 con 0x7fb7240a98f0 2024-03-25T18:34:46.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.103+0000 7fb71a7fc640 1 -- 172.21.15.99:0/4207110946 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7fb720052310 con 0x7fb7240a98f0 2024-03-25T18:34:46.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.103+0000 7fb71a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:46.104 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.103+0000 7fb71a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:46.104 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:34:46.104 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:46.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.105+0000 7fb72c19d640 10 librados: watch_flush enter 2024-03-25T18:34:46.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.105+0000 7fb72c19d640 10 librados: watch_flush exit 2024-03-25T18:34:46.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.105+0000 7fb72c19d640 20 client.4651.objecter shutdown clearing up homeless session... 2024-03-25T18:34:46.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.105+0000 7fb72c19d640 10 client.4651.objecter successfully canceled tick 2024-03-25T18:34:46.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.105+0000 7fb72c19d640 1 -- 172.21.15.99:0/4207110946 >> v1:172.21.15.190:6800/4220409699 conn(0x7fb70806dfb0 legacy=0x7fb708070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.105+0000 7fb72c19d640 10 monclient: shutdown 2024-03-25T18:34:46.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.105+0000 7fb72c19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:46.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.105+0000 7fb72c19d640 1 -- 172.21.15.99:0/4207110946 >> v1:172.21.15.190:6789/0 conn(0x7fb7240a98f0 legacy=0x7fb7241d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.105+0000 7fb72c19d640 1 -- 172.21.15.99:0/4207110946 shutdown_connections 2024-03-25T18:34:46.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.105+0000 7fb72c19d640 1 -- 172.21.15.99:0/4207110946 >> 172.21.15.99:0/4207110946 conn(0x7fb7240b7ea0 msgr2=0x7fb724074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:46.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.106+0000 7fb72c19d640 1 -- 172.21.15.99:0/4207110946 shutdown_connections 2024-03-25T18:34:46.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.106+0000 7fb72c19d640 1 -- 172.21.15.99:0/4207110946 wait complete. 2024-03-25T18:34:46.108 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:46.106+0000 7fb72c19d640 1 librados: shutdown 2024-03-25T18:34:46.118 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:52.924 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:53.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.024+0000 7f2d70649640 10 monclient: get_monmap_and_config 2024-03-25T18:34:53.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.024+0000 7f2d70649640 10 monclient: build_initial_monmap 2024-03-25T18:34:53.025 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.024+0000 7f2d70649640 10 monclient: monmap: 2024-03-25T18:34:53.025 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:34:53.025 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:53.026 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:34:53.025580+0000 2024-03-25T18:34:53.026 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:34:53.025580+0000 2024-03-25T18:34:53.026 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:53.026 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:53.026 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:34:53.026 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:34:53.026 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:34:53.026 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:53.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.025+0000 7f2d70649640 1 Processor -- start 2024-03-25T18:34:53.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.025+0000 7f2d70649640 1 -- start start 2024-03-25T18:34:53.027 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.025+0000 7f2d70649640 10 monclient: init 2024-03-25T18:34:53.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.025+0000 7f2d70649640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:53.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.025+0000 7f2d70649640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:53.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.025+0000 7f2d70649640 10 monclient(hunting): picked mon.noname-a con 0x7f2d6806c380 addr v1:172.21.15.99:6789/0 2024-03-25T18:34:53.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.025+0000 7f2d70649640 10 monclient(hunting): picked mon.noname-b con 0x7f2d680b12d0 addr v1:172.21.15.99:6790/0 2024-03-25T18:34:53.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.025+0000 7f2d70649640 10 monclient(hunting): picked mon.noname-c con 0x7f2d680aedf0 addr v1:172.21.15.190:6789/0 2024-03-25T18:34:53.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.025+0000 7f2d70649640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d680b16f0 con 0x7f2d6806c380 2024-03-25T18:34:53.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.025+0000 7f2d70649640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d680adfe0 con 0x7f2d680aedf0 2024-03-25T18:34:53.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.026+0000 7f2d70649640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d680add00 con 0x7f2d680b12d0 2024-03-25T18:34:53.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.026+0000 7f2d70649640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:53.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.026+0000 7f2d70649640 10 monclient(hunting): authenticate will time out at 7911.193894s 2024-03-25T18:34:53.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.026+0000 7f2d6dbe4640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f2d680b12d0 0x7f2d680ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:46116/0 (socket says 172.21.15.99:46116) 2024-03-25T18:34:53.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.026+0000 7f2d6dbe4640 1 -- 172.21.15.99:0/1811960989 learned_addr learned my addr 172.21.15.99:0/1811960989 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:34:53.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.039+0000 7f2d6d3e3640 1 -- 172.21.15.99:0/1811960989 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570456376 0 0) 0x7f2d680adfe0 con 0x7f2d680aedf0 2024-03-25T18:34:53.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.039+0000 7f2d6d3e3640 10 monclient(hunting): _negotiate 2024-03-25T18:34:53.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.039+0000 7f2d6d3e3640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:53.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.039+0000 7f2d6d3e3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:53.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.039+0000 7f2d6d3e3640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:34:53.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.039+0000 7f2d6d3e3640 1 -- 172.21.15.99:0/1811960989 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d54003520 con 0x7f2d680aedf0 2024-03-25T18:34:53.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.046+0000 7f2d6d3e3640 1 -- 172.21.15.99:0/1811960989 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825917770 0 0) 0x7f2d680b16f0 con 0x7f2d6806c380 2024-03-25T18:34:53.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.046+0000 7f2d6d3e3640 10 monclient(hunting): _negotiate 2024-03-25T18:34:53.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.046+0000 7f2d6d3e3640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:53.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.046+0000 7f2d6d3e3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:53.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.046+0000 7f2d6d3e3640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:34:53.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.046+0000 7f2d6d3e3640 1 -- 172.21.15.99:0/1811960989 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d680adfe0 con 0x7f2d6806c380 2024-03-25T18:34:53.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.058+0000 7f2d6d3e3640 1 -- 172.21.15.99:0/1811960989 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844399611 0 0) 0x7f2d680add00 con 0x7f2d680b12d0 2024-03-25T18:34:53.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.058+0000 7f2d6d3e3640 10 monclient(hunting): _negotiate 2024-03-25T18:34:53.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.058+0000 7f2d6d3e3640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:53.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.058+0000 7f2d6d3e3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:53.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.058+0000 7f2d6d3e3640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:34:53.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.058+0000 7f2d6d3e3640 1 -- 172.21.15.99:0/1811960989 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d680b16f0 con 0x7f2d680b12d0 2024-03-25T18:34:53.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.066+0000 7f2d6d3e3640 1 -- 172.21.15.99:0/1811960989 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2854546061 0 0) 0x7f2d680adfe0 con 0x7f2d6806c380 2024-03-25T18:34:53.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.066+0000 7f2d6d3e3640 1 -- 172.21.15.99:0/1811960989 >> v1:172.21.15.99:6790/0 conn(0x7f2d680b12d0 legacy=0x7f2d680ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:53.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.066+0000 7f2d6d3e3640 1 -- 172.21.15.99:0/1811960989 >> v1:172.21.15.190:6789/0 conn(0x7f2d680aedf0 legacy=0x7f2d6814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:53.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.066+0000 7f2d6d3e3640 10 monclient: _finish_hunting 0 2024-03-25T18:34:53.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.066+0000 7f2d6d3e3640 1 monclient: found mon.noname-a 2024-03-25T18:34:53.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.066+0000 7f2d6d3e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:53.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.066+0000 7f2d6d3e3640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:34:53.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.066+0000 7f2d6d3e3640 1 -- 172.21.15.99:0/1811960989 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d680ae2c0 con 0x7f2d6806c380 2024-03-25T18:34:53.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.067+0000 7f2d6d3e3640 10 monclient: _finish_auth 0 2024-03-25T18:34:53.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.067+0000 7f2d6d3e3640 10 monclient: _check_auth_tickets 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.067+0000 7f2d6d3e3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.067+0000 7f2d6d3e3640 1 -- 172.21.15.99:0/1811960989 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2d64001670 con 0x7f2d6806c380 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.067+0000 7f2d70649640 5 monclient: authenticate success, global_id 4644 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.067+0000 7f2d70649640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.067+0000 7f2d6d3e3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.067+0000 7f2d6d3e3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.067+0000 7f2d6d3e3640 10 monclient: dump: 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:53.069 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:53.070 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:53.070 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:53.070 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.067+0000 7f2d70649640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:53.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.078+0000 7f2d6d3e3640 1 -- 172.21.15.99:0/1811960989 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d640017d0 con 0x7f2d6806c380 2024-03-25T18:34:53.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.078+0000 7f2d6d3e3640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:53.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.078+0000 7f2d6d3e3640 1 -- 172.21.15.99:0/1811960989 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2d64001b60 con 0x7f2d6806c380 2024-03-25T18:34:53.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.078+0000 7f2d70649640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:53.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.078+0000 7f2d70649640 4 set_mon_vals no callback set 2024-03-25T18:34:53.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.078+0000 7f2d6d3e3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:53.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.078+0000 7f2d6d3e3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:53.079 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.078+0000 7f2d6d3e3640 10 monclient: dump: 2024-03-25T18:34:53.079 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:53.080 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:53.080 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:53.080 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:53.080 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:53.080 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:53.080 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:53.080 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:53.080 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:53.080 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:53.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.079+0000 7f2d70649640 10 monclient: shutdown 2024-03-25T18:34:53.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.079+0000 7f2d70649640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:53.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.079+0000 7f2d70649640 1 -- 172.21.15.99:0/1811960989 >> v1:172.21.15.99:6789/0 conn(0x7f2d6806c380 legacy=0x7f2d680b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:53.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.079+0000 7f2d70649640 1 -- 172.21.15.99:0/1811960989 shutdown_connections 2024-03-25T18:34:53.081 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.079+0000 7f2d70649640 1 -- 172.21.15.99:0/1811960989 >> 172.21.15.99:0/1811960989 conn(0x7f2d680b5d10 msgr2=0x7f2d680b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:53.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.079+0000 7f2d70649640 1 -- 172.21.15.99:0/1811960989 shutdown_connections 2024-03-25T18:34:53.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.079+0000 7f2d70649640 1 -- 172.21.15.99:0/1811960989 wait complete. 2024-03-25T18:34:53.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.080+0000 7f2d70649640 10 monclient: build_initial_monmap 2024-03-25T18:34:53.082 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.080+0000 7f2d70649640 10 monclient: monmap: 2024-03-25T18:34:53.082 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:34:53.082 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:53.082 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:34:53.082 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:34:53.082 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:53.082 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:34:53.082 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:34:53.082 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:34:53.082 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:34:53.083 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:53.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.080+0000 7f2d70649640 1 librados: starting msgr at 2024-03-25T18:34:53.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.080+0000 7f2d70649640 1 librados: starting objecter 2024-03-25T18:34:53.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.080+0000 7f2d70649640 1 Processor -- start 2024-03-25T18:34:53.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.080+0000 7f2d70649640 1 -- start start 2024-03-25T18:34:53.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.080+0000 7f2d70649640 1 librados: setting wanted keys 2024-03-25T18:34:53.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.080+0000 7f2d70649640 1 librados: calling monclient init 2024-03-25T18:34:53.083 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.080+0000 7f2d70649640 10 monclient: init 2024-03-25T18:34:53.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.081+0000 7f2d70649640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:53.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.081+0000 7f2d70649640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:53.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.081+0000 7f2d70649640 10 monclient(hunting): picked mon.noname-c con 0x7f2d6806c380 addr v1:172.21.15.99:6790/0 2024-03-25T18:34:53.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.081+0000 7f2d70649640 10 monclient(hunting): picked mon.noname-b con 0x7f2d680aedf0 addr v1:172.21.15.190:6789/0 2024-03-25T18:34:53.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.081+0000 7f2d70649640 10 monclient(hunting): picked mon.noname-a con 0x7f2d680b12d0 addr v1:172.21.15.99:6789/0 2024-03-25T18:34:53.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.081+0000 7f2d70649640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6806c8e0 con 0x7f2d680b12d0 2024-03-25T18:34:53.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.081+0000 7f2d70649640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d681d83a0 con 0x7f2d680aedf0 2024-03-25T18:34:53.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.081+0000 7f2d70649640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d681c25f0 con 0x7f2d6806c380 2024-03-25T18:34:53.084 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.081+0000 7f2d70649640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:53.085 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.081+0000 7f2d70649640 10 monclient(hunting): authenticate will time out at 7911.249549s 2024-03-25T18:34:53.090 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.089+0000 7f2d6dbe4640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f2d680aedf0 0x7f2d681c1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:44776/0 (socket says 172.21.15.99:44776) 2024-03-25T18:34:53.090 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.089+0000 7f2d6dbe4640 1 -- 172.21.15.99:0/3407020421 learned_addr learned my addr 172.21.15.99:0/3407020421 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:34:53.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.090+0000 7f2d5effd640 10 client.?.objecter ms_handle_connect 0x7f2d6806c380 2024-03-25T18:34:53.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.090+0000 7f2d5effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:53.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.092+0000 7f2d5effd640 10 client.?.objecter ms_handle_connect 0x7f2d680aedf0 2024-03-25T18:34:53.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.092+0000 7f2d5effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:53.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.094+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946593106 0 0) 0x7f2d681c25f0 con 0x7f2d6806c380 2024-03-25T18:34:53.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.094+0000 7f2d5effd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:53.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.094+0000 7f2d5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:53.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.094+0000 7f2d5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:53.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.094+0000 7f2d5effd640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:34:53.096 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.094+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d4c004eb0 con 0x7f2d6806c380 2024-03-25T18:34:53.100 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.099+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505683474 0 0) 0x7f2d681d83a0 con 0x7f2d680aedf0 2024-03-25T18:34:53.100 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.099+0000 7f2d5effd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:53.100 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.099+0000 7f2d5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:53.101 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.099+0000 7f2d5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:53.101 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.099+0000 7f2d5effd640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:34:53.101 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.099+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d681c25f0 con 0x7f2d680aedf0 2024-03-25T18:34:53.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.109+0000 7f2d5effd640 10 client.?.objecter ms_handle_connect 0x7f2d680b12d0 2024-03-25T18:34:53.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.109+0000 7f2d5effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:53.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.110+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 899239817 0 0) 0x7f2d681c25f0 con 0x7f2d680aedf0 2024-03-25T18:34:53.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.110+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d681d83a0 con 0x7f2d680aedf0 2024-03-25T18:34:53.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.110+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2d58001850 con 0x7f2d680aedf0 2024-03-25T18:34:53.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.110+0000 7f2d5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:53.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.110+0000 7f2d5effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:53.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.110+0000 7f2d5effd640 10 monclient(hunting): dump: 2024-03-25T18:34:53.112 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:53.112 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:53.112 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:53.112 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:53.112 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:53.112 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:53.112 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:53.112 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:53.112 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:53.112 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:53.112 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.110+0000 7f2d5effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:53.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.112+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2598135312 0 0) 0x7f2d4c004eb0 con 0x7f2d6806c380 2024-03-25T18:34:53.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.112+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d681c25f0 con 0x7f2d6806c380 2024-03-25T18:34:53.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.112+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2d64001ae0 con 0x7f2d6806c380 2024-03-25T18:34:53.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.112+0000 7f2d5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:53.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.112+0000 7f2d5effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:53.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.112+0000 7f2d5effd640 10 monclient(hunting): dump: 2024-03-25T18:34:53.113 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:53.113 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:53.113 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:53.113 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:53.113 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:53.113 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:53.114 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:53.114 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:53.114 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:53.114 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:53.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2400703028 0 0) 0x7f2d681c25f0 con 0x7f2d6806c380 2024-03-25T18:34:53.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 >> v1:172.21.15.190:6789/0 conn(0x7f2d680aedf0 legacy=0x7f2d681c1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:53.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 >> v1:172.21.15.99:6789/0 conn(0x7f2d680b12d0 legacy=0x7f2d681c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d5effd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d5effd640 1 monclient: found mon.c 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d5effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d681c0ef0 con 0x7f2d6806c380 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d5effd640 10 monclient: _finish_auth 0 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d5effd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d70649640 5 monclient: authenticate success, global_id 4652 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d70649640 10 monclient: _renew_subs 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d70649640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d70649640 1 -- 172.21.15.99:0/3407020421 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d681c16d0 con 0x7f2d6806c380 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d70649640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d70649640 10 monclient: _renew_subs 2024-03-25T18:34:53.123 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d70649640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:53.124 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d70649640 1 -- 172.21.15.99:0/3407020421 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d681d7660 con 0x7f2d6806c380 2024-03-25T18:34:53.124 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.119+0000 7f2d70649640 1 librados: init done 2024-03-25T18:34:53.124 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.120+0000 7f2d70649640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:53.124 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.120+0000 7f2d70649640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:53.124 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.120+0000 7f2d70649640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:53.124 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.120+0000 7f2d70649640 1 -- 172.21.15.99:0/3407020421 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d681c41e0 con 0x7f2d6806c380 2024-03-25T18:34:53.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.339+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d640038a0 con 0x7f2d6806c380 2024-03-25T18:34:53.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.339+0000 7f2d5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:53.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.339+0000 7f2d6cbe2640 4 set_mon_vals no callback set 2024-03-25T18:34:53.415 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.414+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f2d64003bb0 con 0x7f2d6806c380 2024-03-25T18:34:53.415 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.414+0000 7f2d5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:53.415 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.414+0000 7f2d5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:53.415 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.414+0000 7f2d5effd640 10 monclient: dump: 2024-03-25T18:34:53.416 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:34:53.416 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:34:53.416 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:53.416 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:34:53.416 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:53.416 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:34:53.416 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:34:53.416 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:34:53.416 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:34:53.416 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:34:53.417 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.415+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f2d64019b80 con 0x7f2d6806c380 2024-03-25T18:34:53.417 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.416+0000 7f2d6dbe4640 10 monclient: get_auth_request con 0x7f2d4c06ddc0 auth_method 0 2024-03-25T18:34:53.417 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.416+0000 7f2d5effd640 10 client.4652.objecter ms_handle_connect 0x7f2d4c06ddc0 2024-03-25T18:34:53.422 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.421+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f2d64053720 con 0x7f2d6806c380 2024-03-25T18:34:53.422 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.421+0000 7f2d5effd640 10 client.4652.objecter ms_dispatch 0x7f2d68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:53.422 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.421+0000 7f2d5effd640 3 client.4652.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:53.422 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.421+0000 7f2d5effd640 3 client.4652.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:53.422 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.421+0000 7f2d5effd640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-25T18:34:53.422 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.421+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d6404f060 con 0x7f2d6806c380 2024-03-25T18:34:53.422 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.421+0000 7f2d5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:53.423 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.421+0000 7f2d5effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:53.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.676+0000 7f2d70649640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:53.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.676+0000 7f2d70649640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:53.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.676+0000 7f2d70649640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:34:53.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.676+0000 7f2d70649640 1 -- 172.21.15.99:0/3407020421 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d680627b0 con 0x7f2d6806c380 2024-03-25T18:34:53.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.696+0000 7f2d5effd640 1 -- 172.21.15.99:0/3407020421 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f2d64051160 con 0x7f2d6806c380 2024-03-25T18:34:53.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.696+0000 7f2d5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:53.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.696+0000 7f2d5effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:53.697 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:34:53.697 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:53.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.698+0000 7f2d70649640 10 librados: watch_flush enter 2024-03-25T18:34:53.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.698+0000 7f2d70649640 10 librados: watch_flush exit 2024-03-25T18:34:53.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.698+0000 7f2d70649640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-25T18:34:53.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.698+0000 7f2d70649640 10 client.4652.objecter successfully canceled tick 2024-03-25T18:34:53.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.698+0000 7f2d70649640 1 -- 172.21.15.99:0/3407020421 >> v1:172.21.15.190:6800/4220409699 conn(0x7f2d4c06ddc0 legacy=0x7f2d4c070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:53.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.698+0000 7f2d70649640 10 monclient: shutdown 2024-03-25T18:34:53.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.698+0000 7f2d70649640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:53.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.698+0000 7f2d70649640 1 -- 172.21.15.99:0/3407020421 >> v1:172.21.15.99:6790/0 conn(0x7f2d6806c380 legacy=0x7f2d681c4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:53.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.698+0000 7f2d70649640 1 -- 172.21.15.99:0/3407020421 shutdown_connections 2024-03-25T18:34:53.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.698+0000 7f2d70649640 1 -- 172.21.15.99:0/3407020421 >> 172.21.15.99:0/3407020421 conn(0x7f2d680b5d10 msgr2=0x7f2d681cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:53.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.698+0000 7f2d70649640 1 -- 172.21.15.99:0/3407020421 shutdown_connections 2024-03-25T18:34:53.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.698+0000 7f2d70649640 1 -- 172.21.15.99:0/3407020421 wait complete. 2024-03-25T18:34:53.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:34:53.699+0000 7f2d70649640 1 librados: shutdown 2024-03-25T18:34:53.711 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:55.156 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:09:55.749 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:55.199 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:09:55.757 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:55.201 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:09:55.760 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:55.202 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:09:55.795 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:55.202 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:09:55.795 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:55.203 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:09:55.796 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:55.224 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:09:55.783 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:55.247 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:09:55.840 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:55.250 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:09:55.843 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:00.517 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:00.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.631+0000 7f591ab3d640 10 monclient: get_monmap_and_config 2024-03-25T18:35:00.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.631+0000 7f591ab3d640 10 monclient: build_initial_monmap 2024-03-25T18:35:00.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.631+0000 7f591ab3d640 10 monclient: monmap: 2024-03-25T18:35:00.633 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:00.633 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:00.633 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:35:00.633001+0000 2024-03-25T18:35:00.633 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:35:00.633001+0000 2024-03-25T18:35:00.633 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:00.633 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:00.633 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:00.633 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:35:00.633 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:35:00.633 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:00.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.632+0000 7f591ab3d640 1 Processor -- start 2024-03-25T18:35:00.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.632+0000 7f591ab3d640 1 -- start start 2024-03-25T18:35:00.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.632+0000 7f591ab3d640 10 monclient: init 2024-03-25T18:35:00.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.633+0000 7f591ab3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:00.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.633+0000 7f591ab3d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:00.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.633+0000 7f591ab3d640 10 monclient(hunting): picked mon.noname-c con 0x7f59140a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:00.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.633+0000 7f591ab3d640 10 monclient(hunting): picked mon.noname-b con 0x7f591413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:00.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.633+0000 7f591ab3d640 10 monclient(hunting): picked mon.noname-a con 0x7f59141430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:00.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.633+0000 7f591ab3d640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5914142520 con 0x7f59141430c0 2024-03-25T18:35:00.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.633+0000 7f591ab3d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59140b28a0 con 0x7f59140a98f0 2024-03-25T18:35:00.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.633+0000 7f591ab3d640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59140b2a30 con 0x7f591413f710 2024-03-25T18:35:00.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.633+0000 7f591ab3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:00.636 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.633+0000 7f591ab3d640 10 monclient(hunting): authenticate will time out at 7918.801507s 2024-03-25T18:35:00.641 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.640+0000 7f59188d9640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f59140a98f0 0x7f59140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:44372/0 (socket says 172.21.15.99:44372) 2024-03-25T18:35:00.641 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.640+0000 7f59188d9640 1 -- 172.21.15.99:0/3852063719 learned_addr learned my addr 172.21.15.99:0/3852063719 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:00.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.656+0000 7f59137fe640 1 -- 172.21.15.99:0/3852063719 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292746134 0 0) 0x7f59140b28a0 con 0x7f59140a98f0 2024-03-25T18:35:00.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.656+0000 7f59137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:00.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.656+0000 7f59137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:00.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.656+0000 7f59137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:00.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.656+0000 7f59137fe640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:35:00.657 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.656+0000 7f59137fe640 1 -- 172.21.15.99:0/3852063719 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5904003520 con 0x7f59140a98f0 2024-03-25T18:35:00.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.667+0000 7f59137fe640 1 -- 172.21.15.99:0/3852063719 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644116805 0 0) 0x7f59140b2a30 con 0x7f591413f710 2024-03-25T18:35:00.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.667+0000 7f59137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:00.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.667+0000 7f59137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:00.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.667+0000 7f59137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:00.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.667+0000 7f59137fe640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:35:00.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.667+0000 7f59137fe640 1 -- 172.21.15.99:0/3852063719 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59140b28a0 con 0x7f591413f710 2024-03-25T18:35:00.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.668+0000 7f59137fe640 1 -- 172.21.15.99:0/3852063719 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294642729 0 0) 0x7f5914142520 con 0x7f59141430c0 2024-03-25T18:35:00.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.668+0000 7f59137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:00.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.668+0000 7f59137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:00.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.668+0000 7f59137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:00.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.668+0000 7f59137fe640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:35:00.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.668+0000 7f59137fe640 1 -- 172.21.15.99:0/3852063719 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59140b2a30 con 0x7f59141430c0 2024-03-25T18:35:00.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.673+0000 7f59137fe640 1 -- 172.21.15.99:0/3852063719 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3428089764 0 0) 0x7f5904003520 con 0x7f59140a98f0 2024-03-25T18:35:00.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.673+0000 7f59137fe640 1 -- 172.21.15.99:0/3852063719 >> v1:172.21.15.99:6790/0 conn(0x7f591413f710 legacy=0x7f59141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:00.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.673+0000 7f59137fe640 1 -- 172.21.15.99:0/3852063719 >> v1:172.21.15.99:6789/0 conn(0x7f59141430c0 legacy=0x7f591414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:00.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.673+0000 7f59137fe640 10 monclient: _finish_hunting 0 2024-03-25T18:35:00.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.673+0000 7f59137fe640 1 monclient: found mon.noname-c 2024-03-25T18:35:00.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.673+0000 7f59137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.674+0000 7f59137fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.674+0000 7f59137fe640 1 -- 172.21.15.99:0/3852063719 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59140b44c0 con 0x7f59140a98f0 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.674+0000 7f59137fe640 10 monclient: _finish_auth 0 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.674+0000 7f59137fe640 10 monclient: _check_auth_tickets 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.674+0000 7f59137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.674+0000 7f59137fe640 1 -- 172.21.15.99:0/3852063719 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5908001690 con 0x7f59140a98f0 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.674+0000 7f59137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.674+0000 7f59137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.674+0000 7f59137fe640 10 monclient: dump: 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:00.678 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:00.679 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:00.679 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:00.679 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:00.679 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:00.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.674+0000 7f591ab3d640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:35:00.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.674+0000 7f591ab3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:00.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.696+0000 7f59137fe640 1 -- 172.21.15.99:0/3852063719 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59080017f0 con 0x7f59140a98f0 2024-03-25T18:35:00.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.696+0000 7f59137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:00.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.696+0000 7f59137fe640 1 -- 172.21.15.99:0/3852063719 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5908001c00 con 0x7f59140a98f0 2024-03-25T18:35:00.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.696+0000 7f591ab3d640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:00.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.696+0000 7f591ab3d640 4 set_mon_vals no callback set 2024-03-25T18:35:00.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.696+0000 7f59137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:00.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.696+0000 7f59137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:00.697 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.696+0000 7f59137fe640 10 monclient: dump: 2024-03-25T18:35:00.698 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:00.698 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:00.698 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:00.698 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:00.698 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:00.698 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:00.698 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:00.698 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:00.698 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:00.698 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:00.699 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.697+0000 7f591ab3d640 10 monclient: shutdown 2024-03-25T18:35:00.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.697+0000 7f591ab3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:00.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.697+0000 7f591ab3d640 1 -- 172.21.15.99:0/3852063719 >> v1:172.21.15.190:6789/0 conn(0x7f59140a98f0 legacy=0x7f59140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:00.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.697+0000 7f591ab3d640 1 -- 172.21.15.99:0/3852063719 shutdown_connections 2024-03-25T18:35:00.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.697+0000 7f591ab3d640 1 -- 172.21.15.99:0/3852063719 >> 172.21.15.99:0/3852063719 conn(0x7f59140b7ea0 msgr2=0x7f59140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:00.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.697+0000 7f591ab3d640 1 -- 172.21.15.99:0/3852063719 shutdown_connections 2024-03-25T18:35:00.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.697+0000 7f591ab3d640 1 -- 172.21.15.99:0/3852063719 wait complete. 2024-03-25T18:35:00.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.698+0000 7f591ab3d640 10 monclient: build_initial_monmap 2024-03-25T18:35:00.700 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.698+0000 7f591ab3d640 10 monclient: monmap: 2024-03-25T18:35:00.700 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:00.700 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:00.700 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:35:00.700 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:35:00.700 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:00.700 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:35:00.701 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:00.701 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:35:00.701 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:35:00.701 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:00.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.698+0000 7f591ab3d640 1 librados: starting msgr at 2024-03-25T18:35:00.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.698+0000 7f591ab3d640 1 librados: starting objecter 2024-03-25T18:35:00.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.698+0000 7f591ab3d640 1 Processor -- start 2024-03-25T18:35:00.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.698+0000 7f591ab3d640 1 -- start start 2024-03-25T18:35:00.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.698+0000 7f591ab3d640 1 librados: setting wanted keys 2024-03-25T18:35:00.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.698+0000 7f591ab3d640 1 librados: calling monclient init 2024-03-25T18:35:00.701 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.698+0000 7f591ab3d640 10 monclient: init 2024-03-25T18:35:00.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.699+0000 7f591ab3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:00.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.699+0000 7f591ab3d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:00.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.699+0000 7f591ab3d640 10 monclient(hunting): picked mon.noname-a con 0x7f59140a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:00.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.699+0000 7f591ab3d640 10 monclient(hunting): picked mon.noname-b con 0x7f591413f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:00.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.699+0000 7f591ab3d640 10 monclient(hunting): picked mon.noname-c con 0x7f59141430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:00.702 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.699+0000 7f591ab3d640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59140accf0 con 0x7f59140a98f0 2024-03-25T18:35:00.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.699+0000 7f591ab3d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59140abe70 con 0x7f591413f710 2024-03-25T18:35:00.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.699+0000 7f591ab3d640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59140ac150 con 0x7f59141430c0 2024-03-25T18:35:00.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.699+0000 7f591ab3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:00.703 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.699+0000 7f591ab3d640 10 monclient(hunting): authenticate will time out at 7918.867804s 2024-03-25T18:35:00.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.711+0000 7f5913fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f591413f710 0x7f59140ac5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:44428/0 (socket says 172.21.15.99:44428) 2024-03-25T18:35:00.712 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.711+0000 7f5913fff640 1 -- 172.21.15.99:0/2623270860 learned_addr learned my addr 172.21.15.99:0/2623270860 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:00.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.714+0000 7f59117fa640 10 client.?.objecter ms_handle_connect 0x7f591413f710 2024-03-25T18:35:00.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.714+0000 7f59117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:00.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.723+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2207533418 0 0) 0x7f59140abe70 con 0x7f591413f710 2024-03-25T18:35:00.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.723+0000 7f59117fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:00.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.723+0000 7f59117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:00.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.723+0000 7f59117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:00.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.723+0000 7f59117fa640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:35:00.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.723+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f8004eb0 con 0x7f591413f710 2024-03-25T18:35:00.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.724+0000 7f59117fa640 10 client.?.objecter ms_handle_connect 0x7f59140a98f0 2024-03-25T18:35:00.724 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.724+0000 7f59117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:00.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.731+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490469573 0 0) 0x7f59140accf0 con 0x7f59140a98f0 2024-03-25T18:35:00.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.731+0000 7f59117fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:00.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.731+0000 7f59117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:00.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.731+0000 7f59117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:00.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.731+0000 7f59117fa640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:35:00.732 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.731+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59140abe70 con 0x7f59140a98f0 2024-03-25T18:35:00.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.732+0000 7f59117fa640 10 client.?.objecter ms_handle_connect 0x7f59141430c0 2024-03-25T18:35:00.733 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.732+0000 7f59117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:00.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.752+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3492144863 0 0) 0x7f58f8004eb0 con 0x7f591413f710 2024-03-25T18:35:00.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.752+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59140accf0 con 0x7f591413f710 2024-03-25T18:35:00.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.752+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5900001a30 con 0x7f591413f710 2024-03-25T18:35:00.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.752+0000 7f59117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:00.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.752+0000 7f59117fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:00.753 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.752+0000 7f59117fa640 10 monclient(hunting): dump: 2024-03-25T18:35:00.753 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:00.753 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:00.754 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:00.754 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:00.754 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:00.754 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:00.754 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:00.754 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:00.754 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:00.754 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:00.754 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.752+0000 7f59117fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:00.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.761+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4265543807 0 0) 0x7f59140abe70 con 0x7f59140a98f0 2024-03-25T18:35:00.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.761+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58f8004eb0 con 0x7f59140a98f0 2024-03-25T18:35:00.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.761+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5908001b00 con 0x7f59140a98f0 2024-03-25T18:35:00.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.761+0000 7f59117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:00.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.761+0000 7f59117fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:00.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.761+0000 7f59117fa640 10 monclient(hunting): dump: 2024-03-25T18:35:00.763 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:00.763 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:00.763 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:00.763 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:00.763 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:00.763 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:00.763 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:00.763 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:00.763 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:00.763 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:00.763 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.762+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447106609 0 0) 0x7f59140ac150 con 0x7f59141430c0 2024-03-25T18:35:00.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.762+0000 7f59117fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:00.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.762+0000 7f59117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:00.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.762+0000 7f59117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:00.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.762+0000 7f59117fa640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:35:00.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.762+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59140abe70 con 0x7f59141430c0 2024-03-25T18:35:00.774 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.772+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2330222655 0 0) 0x7f59140accf0 con 0x7f591413f710 2024-03-25T18:35:00.774 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.772+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 >> v1:172.21.15.99:6790/0 conn(0x7f59141430c0 legacy=0x7f59141e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:00.774 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.772+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 >> v1:172.21.15.99:6789/0 conn(0x7f59140a98f0 legacy=0x7f59141389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:00.774 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.772+0000 7f59117fa640 10 monclient: _finish_hunting 0 2024-03-25T18:35:00.774 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.772+0000 7f59117fa640 1 monclient: found mon.b 2024-03-25T18:35:00.774 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.772+0000 7f59117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:00.774 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.772+0000 7f59117fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:00.774 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.772+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59141392c0 con 0x7f591413f710 2024-03-25T18:35:00.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.773+0000 7f59117fa640 10 monclient: _finish_auth 0 2024-03-25T18:35:00.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.773+0000 7f59117fa640 10 monclient: _check_auth_tickets 2024-03-25T18:35:00.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.773+0000 7f59117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:00.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.773+0000 7f591ab3d640 5 monclient: authenticate success, global_id 4663 2024-03-25T18:35:00.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.773+0000 7f591ab3d640 10 monclient: _renew_subs 2024-03-25T18:35:00.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.773+0000 7f591ab3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:00.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.773+0000 7f591ab3d640 1 -- 172.21.15.99:0/2623270860 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5914137e50 con 0x7f591413f710 2024-03-25T18:35:00.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.773+0000 7f591ab3d640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:00.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.773+0000 7f591ab3d640 10 monclient: _renew_subs 2024-03-25T18:35:00.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.773+0000 7f591ab3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:00.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.773+0000 7f591ab3d640 1 -- 172.21.15.99:0/2623270860 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59141368f0 con 0x7f591413f710 2024-03-25T18:35:00.775 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.773+0000 7f591ab3d640 1 librados: init done 2024-03-25T18:35:00.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.774+0000 7f591ab3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:00.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.774+0000 7f591ab3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:00.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.774+0000 7f591ab3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:00.776 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:00.774+0000 7f591ab3d640 1 -- 172.21.15.99:0/2623270860 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5914136ba0 con 0x7f591413f710 2024-03-25T18:35:01.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.054+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5900001c90 con 0x7f591413f710 2024-03-25T18:35:01.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.054+0000 7f59117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:01.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.054+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5900003a60 con 0x7f591413f710 2024-03-25T18:35:01.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.054+0000 7f59117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:01.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.054+0000 7f59117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:01.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.054+0000 7f59117fa640 10 monclient: dump: 2024-03-25T18:35:01.056 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:01.056 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:01.056 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:01.056 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:01.056 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:01.057 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:01.057 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:01.057 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:01.057 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:01.057 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:01.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.054+0000 7f5912ffd640 4 set_mon_vals no callback set 2024-03-25T18:35:01.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.055+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f5900003c80 con 0x7f591413f710 2024-03-25T18:35:01.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.056+0000 7f59188d9640 10 monclient: get_auth_request con 0x7f58f806df50 auth_method 0 2024-03-25T18:35:01.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.056+0000 7f59117fa640 10 client.4663.objecter ms_handle_connect 0x7f58f806df50 2024-03-25T18:35:01.062 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.061+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f5900054860 con 0x7f591413f710 2024-03-25T18:35:01.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.061+0000 7f59117fa640 10 client.4663.objecter ms_dispatch 0x7f5914000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:01.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.061+0000 7f59117fa640 3 client.4663.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:01.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.061+0000 7f59117fa640 3 client.4663.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:01.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.062+0000 7f59117fa640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T18:35:01.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.062+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5900050370 con 0x7f591413f710 2024-03-25T18:35:01.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.062+0000 7f59117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:01.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.062+0000 7f59117fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:01.306 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.306+0000 7f591ab3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:01.307 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.306+0000 7f591ab3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:01.307 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.306+0000 7f591ab3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:01.307 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.306+0000 7f591ab3d640 1 -- 172.21.15.99:0/2623270860 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5914139100 con 0x7f591413f710 2024-03-25T18:35:01.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.338+0000 7f59117fa640 1 -- 172.21.15.99:0/2623270860 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f59000522a0 con 0x7f591413f710 2024-03-25T18:35:01.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.338+0000 7f59117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:01.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.338+0000 7f59117fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:01.339 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:35:01.339 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:01.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.341+0000 7f591ab3d640 10 librados: watch_flush enter 2024-03-25T18:35:01.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.341+0000 7f591ab3d640 10 librados: watch_flush exit 2024-03-25T18:35:01.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.341+0000 7f591ab3d640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T18:35:01.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.341+0000 7f591ab3d640 10 client.4663.objecter successfully canceled tick 2024-03-25T18:35:01.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.341+0000 7f591ab3d640 1 -- 172.21.15.99:0/2623270860 >> v1:172.21.15.190:6800/4220409699 conn(0x7f58f806df50 legacy=0x7f58f8070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:01.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.341+0000 7f591ab3d640 10 monclient: shutdown 2024-03-25T18:35:01.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.341+0000 7f591ab3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:01.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.341+0000 7f591ab3d640 1 -- 172.21.15.99:0/2623270860 >> v1:172.21.15.190:6789/0 conn(0x7f591413f710 legacy=0x7f59140ac5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:01.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.341+0000 7f591ab3d640 1 -- 172.21.15.99:0/2623270860 shutdown_connections 2024-03-25T18:35:01.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.341+0000 7f591ab3d640 1 -- 172.21.15.99:0/2623270860 >> 172.21.15.99:0/2623270860 conn(0x7f59140b7ea0 msgr2=0x7f5914074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:01.342 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.341+0000 7f591ab3d640 1 -- 172.21.15.99:0/2623270860 shutdown_connections 2024-03-25T18:35:01.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.341+0000 7f591ab3d640 1 -- 172.21.15.99:0/2623270860 wait complete. 2024-03-25T18:35:01.343 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:01.341+0000 7f591ab3d640 1 librados: shutdown 2024-03-25T18:35:01.352 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:08.159 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:08.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.274+0000 7f9a3d881640 10 monclient: get_monmap_and_config 2024-03-25T18:35:08.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.274+0000 7f9a3d881640 10 monclient: build_initial_monmap 2024-03-25T18:35:08.275 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.274+0000 7f9a3d881640 10 monclient: monmap: 2024-03-25T18:35:08.276 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:08.276 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:08.276 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:35:08.275612+0000 2024-03-25T18:35:08.276 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:35:08.275612+0000 2024-03-25T18:35:08.276 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:08.276 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:08.276 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:08.276 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:35:08.276 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:35:08.276 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:08.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.275+0000 7f9a3d881640 1 Processor -- start 2024-03-25T18:35:08.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.275+0000 7f9a3d881640 1 -- start start 2024-03-25T18:35:08.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.275+0000 7f9a3d881640 10 monclient: init 2024-03-25T18:35:08.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.276+0000 7f9a3d881640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:08.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.276+0000 7f9a3d881640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:08.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.276+0000 7f9a3d881640 10 monclient(hunting): picked mon.noname-a con 0x7f9a380a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:08.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.276+0000 7f9a3d881640 10 monclient(hunting): picked mon.noname-c con 0x7f9a3813f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:08.278 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.276+0000 7f9a3d881640 10 monclient(hunting): picked mon.noname-b con 0x7f9a381430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:08.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.276+0000 7f9a3d881640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a38142520 con 0x7f9a380a98f0 2024-03-25T18:35:08.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.276+0000 7f9a3d881640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a380b28a0 con 0x7f9a3813f710 2024-03-25T18:35:08.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.276+0000 7f9a3d881640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a380b2a30 con 0x7f9a381430c0 2024-03-25T18:35:08.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.276+0000 7f9a3d881640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:08.279 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.276+0000 7f9a3d881640 10 monclient(hunting): authenticate will time out at 7926.444321s 2024-03-25T18:35:08.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.289+0000 7f9a36ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f9a380a98f0 0x7f9a380a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45238/0 (socket says 172.21.15.99:45238) 2024-03-25T18:35:08.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.289+0000 7f9a36ffd640 1 -- 172.21.15.99:0/1215279818 learned_addr learned my addr 172.21.15.99:0/1215279818 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:08.305 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.303+0000 7f9a35ffb640 1 -- 172.21.15.99:0/1215279818 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188057330 0 0) 0x7f9a380b28a0 con 0x7f9a3813f710 2024-03-25T18:35:08.305 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.303+0000 7f9a35ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:08.305 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.303+0000 7f9a35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:08.305 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.303+0000 7f9a35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:08.305 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.303+0000 7f9a35ffb640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:35:08.305 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.303+0000 7f9a35ffb640 1 -- 172.21.15.99:0/1215279818 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a20003520 con 0x7f9a3813f710 2024-03-25T18:35:08.310 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 1 -- 172.21.15.99:0/1215279818 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3970492342 0 0) 0x7f9a20003520 con 0x7f9a3813f710 2024-03-25T18:35:08.310 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 1 -- 172.21.15.99:0/1215279818 >> v1:172.21.15.99:6790/0 conn(0x7f9a381430c0 legacy=0x7f9a3814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:08.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 1 -- 172.21.15.99:0/1215279818 >> v1:172.21.15.99:6789/0 conn(0x7f9a380a98f0 legacy=0x7f9a380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:08.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:35:08.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 1 monclient: found mon.noname-c 2024-03-25T18:35:08.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:08.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:35:08.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 1 -- 172.21.15.99:0/1215279818 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a380b44c0 con 0x7f9a3813f710 2024-03-25T18:35:08.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 10 monclient: _finish_auth 0 2024-03-25T18:35:08.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:35:08.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:08.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 1 -- 172.21.15.99:0/1215279818 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9a2c001690 con 0x7f9a3813f710 2024-03-25T18:35:08.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:08.311 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:08.312 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a35ffb640 10 monclient: dump: 2024-03-25T18:35:08.312 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:08.312 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:08.312 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:08.312 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:08.312 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:08.312 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:08.312 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:08.312 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:08.312 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:08.312 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:08.312 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a3d881640 5 monclient: authenticate success, global_id 4666 2024-03-25T18:35:08.312 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.309+0000 7f9a3d881640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:08.334 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.333+0000 7f9a35ffb640 1 -- 172.21.15.99:0/1215279818 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a2c0017f0 con 0x7f9a3813f710 2024-03-25T18:35:08.335 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.333+0000 7f9a35ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:08.335 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.333+0000 7f9a35ffb640 1 -- 172.21.15.99:0/1215279818 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9a2c001c00 con 0x7f9a3813f710 2024-03-25T18:35:08.335 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.333+0000 7f9a35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:08.335 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.333+0000 7f9a35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:08.335 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.333+0000 7f9a35ffb640 10 monclient: dump: 2024-03-25T18:35:08.335 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:08.335 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:08.335 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:08.335 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:08.335 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:08.335 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:08.335 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:08.335 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:08.336 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:08.336 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:08.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.333+0000 7f9a3d881640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:08.336 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.333+0000 7f9a3d881640 4 set_mon_vals no callback set 2024-03-25T18:35:08.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.334+0000 7f9a3d881640 10 monclient: shutdown 2024-03-25T18:35:08.337 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.334+0000 7f9a3d881640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:08.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.334+0000 7f9a3d881640 1 -- 172.21.15.99:0/1215279818 >> v1:172.21.15.190:6789/0 conn(0x7f9a3813f710 legacy=0x7f9a381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:08.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.334+0000 7f9a3d881640 1 -- 172.21.15.99:0/1215279818 shutdown_connections 2024-03-25T18:35:08.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.334+0000 7f9a3d881640 1 -- 172.21.15.99:0/1215279818 >> 172.21.15.99:0/1215279818 conn(0x7f9a380b7ea0 msgr2=0x7f9a380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:08.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.334+0000 7f9a3d881640 1 -- 172.21.15.99:0/1215279818 shutdown_connections 2024-03-25T18:35:08.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.334+0000 7f9a3d881640 1 -- 172.21.15.99:0/1215279818 wait complete. 2024-03-25T18:35:08.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.335+0000 7f9a3d881640 10 monclient: build_initial_monmap 2024-03-25T18:35:08.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.335+0000 7f9a3d881640 10 monclient: monmap: 2024-03-25T18:35:08.338 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:08.338 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:08.338 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:35:08.338 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:35:08.338 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:08.339 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:35:08.339 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:08.339 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:35:08.339 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:35:08.339 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:08.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.336+0000 7f9a3d881640 1 librados: starting msgr at 2024-03-25T18:35:08.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.336+0000 7f9a3d881640 1 librados: starting objecter 2024-03-25T18:35:08.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.336+0000 7f9a3d881640 1 Processor -- start 2024-03-25T18:35:08.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.336+0000 7f9a3d881640 1 -- start start 2024-03-25T18:35:08.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.336+0000 7f9a3d881640 1 librados: setting wanted keys 2024-03-25T18:35:08.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.336+0000 7f9a3d881640 1 librados: calling monclient init 2024-03-25T18:35:08.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.336+0000 7f9a3d881640 10 monclient: init 2024-03-25T18:35:08.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.337+0000 7f9a3d881640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:08.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.337+0000 7f9a3d881640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:08.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.337+0000 7f9a3d881640 10 monclient(hunting): picked mon.noname-c con 0x7f9a380a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:08.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.337+0000 7f9a3d881640 10 monclient(hunting): picked mon.noname-b con 0x7f9a3813f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:08.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.337+0000 7f9a3d881640 10 monclient(hunting): picked mon.noname-a con 0x7f9a381430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:08.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.337+0000 7f9a3d881640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a380b1870 con 0x7f9a381430c0 2024-03-25T18:35:08.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.337+0000 7f9a3d881640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a380ac910 con 0x7f9a3813f710 2024-03-25T18:35:08.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.337+0000 7f9a3d881640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a380abb00 con 0x7f9a380a98f0 2024-03-25T18:35:08.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.337+0000 7f9a3d881640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:08.341 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.337+0000 7f9a3d881640 10 monclient(hunting): authenticate will time out at 7926.505750s 2024-03-25T18:35:08.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.348+0000 7f9a377fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f9a381430c0 0x7f9a381e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45290/0 (socket says 172.21.15.99:45290) 2024-03-25T18:35:08.349 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.348+0000 7f9a377fe640 1 -- 172.21.15.99:0/221043736 learned_addr learned my addr 172.21.15.99:0/221043736 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:08.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.349+0000 7f9a1ffff640 10 client.?.objecter ms_handle_connect 0x7f9a381430c0 2024-03-25T18:35:08.350 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.349+0000 7f9a1ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:08.356 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.355+0000 7f9a1ffff640 10 client.?.objecter ms_handle_connect 0x7f9a380a98f0 2024-03-25T18:35:08.356 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.355+0000 7f9a1ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:08.361 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.360+0000 7f9a1ffff640 10 client.?.objecter ms_handle_connect 0x7f9a3813f710 2024-03-25T18:35:08.361 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.360+0000 7f9a1ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:08.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.360+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944838222 0 0) 0x7f9a380b2a30 con 0x7f9a381430c0 2024-03-25T18:35:08.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.360+0000 7f9a1ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:08.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.360+0000 7f9a1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:08.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.360+0000 7f9a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:08.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.360+0000 7f9a1ffff640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:35:08.362 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.360+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a24004eb0 con 0x7f9a381430c0 2024-03-25T18:35:08.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.368+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789579602 0 0) 0x7f9a380abb00 con 0x7f9a380a98f0 2024-03-25T18:35:08.369 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.368+0000 7f9a1ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:08.370 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.368+0000 7f9a1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:08.370 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.368+0000 7f9a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:08.370 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.368+0000 7f9a1ffff640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:35:08.370 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.368+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a380b2a30 con 0x7f9a380a98f0 2024-03-25T18:35:08.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.370+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903071399 0 0) 0x7f9a380ac910 con 0x7f9a3813f710 2024-03-25T18:35:08.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.370+0000 7f9a1ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:08.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.370+0000 7f9a1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:08.371 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.370+0000 7f9a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:08.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.370+0000 7f9a1ffff640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:35:08.372 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.370+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a380abb00 con 0x7f9a3813f710 2024-03-25T18:35:08.377 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.376+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2491059272 0 0) 0x7f9a24004eb0 con 0x7f9a381430c0 2024-03-25T18:35:08.377 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.376+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a380ac910 con 0x7f9a381430c0 2024-03-25T18:35:08.377 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.376+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9a30001af0 con 0x7f9a381430c0 2024-03-25T18:35:08.377 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.376+0000 7f9a1ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:08.378 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.376+0000 7f9a1ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:08.378 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.376+0000 7f9a1ffff640 10 monclient(hunting): dump: 2024-03-25T18:35:08.378 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:08.378 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:08.378 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:08.378 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:08.378 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:08.378 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:08.378 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:08.378 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:08.378 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:08.378 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:08.378 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.376+0000 7f9a1ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:08.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4121012941 0 0) 0x7f9a380ac910 con 0x7f9a381430c0 2024-03-25T18:35:08.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 >> v1:172.21.15.99:6790/0 conn(0x7f9a380a98f0 legacy=0x7f9a38138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:08.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 >> v1:172.21.15.190:6789/0 conn(0x7f9a3813f710 legacy=0x7f9a380ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:08.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a1ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:08.383 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a1ffff640 1 monclient: found mon.a 2024-03-25T18:35:08.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:08.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a1ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:35:08.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a380abde0 con 0x7f9a381430c0 2024-03-25T18:35:08.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a1ffff640 10 monclient: _finish_auth 0 2024-03-25T18:35:08.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a1ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:08.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:08.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a3d881640 5 monclient: authenticate success, global_id 4659 2024-03-25T18:35:08.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a3d881640 10 monclient: _renew_subs 2024-03-25T18:35:08.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a3d881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:35:08.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a3d881640 1 -- 172.21.15.99:0/221043736 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a38138f40 con 0x7f9a381430c0 2024-03-25T18:35:08.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a3d881640 10 client.4659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:08.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a3d881640 10 monclient: _renew_subs 2024-03-25T18:35:08.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a3d881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:35:08.384 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a3d881640 1 -- 172.21.15.99:0/221043736 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a381363f0 con 0x7f9a381430c0 2024-03-25T18:35:08.385 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.382+0000 7f9a3d881640 1 librados: init done 2024-03-25T18:35:08.385 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.383+0000 7f9a3d881640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:08.385 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.383+0000 7f9a3d881640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:08.385 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.383+0000 7f9a3d881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:35:08.385 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.383+0000 7f9a3d881640 1 -- 172.21.15.99:0/221043736 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a380627b0 con 0x7f9a381430c0 2024-03-25T18:35:08.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.685+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a30001fe0 con 0x7f9a381430c0 2024-03-25T18:35:08.686 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.685+0000 7f9a1ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:08.687 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.685+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9a30003b30 con 0x7f9a381430c0 2024-03-25T18:35:08.687 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.685+0000 7f9a1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:08.687 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.685+0000 7f9a1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:08.687 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.685+0000 7f9a1ffff640 10 monclient: dump: 2024-03-25T18:35:08.687 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:08.687 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:08.687 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:08.687 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:08.687 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:08.687 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:08.688 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:08.688 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:08.688 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:08.688 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:08.688 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.685+0000 7f9a357fa640 4 set_mon_vals no callback set 2024-03-25T18:35:08.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.686+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f9a30003d50 con 0x7f9a381430c0 2024-03-25T18:35:08.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.686+0000 7f9a36ffd640 10 monclient: get_auth_request con 0x7f9a2406ddf0 auth_method 0 2024-03-25T18:35:08.689 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.687+0000 7f9a1ffff640 10 client.4659.objecter ms_handle_connect 0x7f9a2406ddf0 2024-03-25T18:35:08.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.693+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f9a30054970 con 0x7f9a381430c0 2024-03-25T18:35:08.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.693+0000 7f9a1ffff640 10 client.4659.objecter ms_dispatch 0x7f9a38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:08.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.693+0000 7f9a1ffff640 3 client.4659.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:08.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.693+0000 7f9a1ffff640 3 client.4659.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:08.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.693+0000 7f9a1ffff640 20 client.4659.objecter dump_active .. 0 homeless 2024-03-25T18:35:08.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.693+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a30050480 con 0x7f9a381430c0 2024-03-25T18:35:08.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.693+0000 7f9a1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:08.695 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.693+0000 7f9a1ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:08.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.948+0000 7f9a3d881640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:08.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.948+0000 7f9a3d881640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:08.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.948+0000 7f9a3d881640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:35:08.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:08.948+0000 7f9a3d881640 1 -- 172.21.15.99:0/221043736 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a381434a0 con 0x7f9a381430c0 2024-03-25T18:35:09.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.033+0000 7f9a1ffff640 1 -- 172.21.15.99:0/221043736 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f9a300523b0 con 0x7f9a381430c0 2024-03-25T18:35:09.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.033+0000 7f9a1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:09.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.033+0000 7f9a1ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:09.035 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:35:09.036 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:09.038 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.035+0000 7f9a3d881640 10 librados: watch_flush enter 2024-03-25T18:35:09.038 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.035+0000 7f9a3d881640 10 librados: watch_flush exit 2024-03-25T18:35:09.038 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.035+0000 7f9a3d881640 20 client.4659.objecter shutdown clearing up homeless session... 2024-03-25T18:35:09.038 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.035+0000 7f9a3d881640 10 client.4659.objecter successfully canceled tick 2024-03-25T18:35:09.038 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.035+0000 7f9a3d881640 1 -- 172.21.15.99:0/221043736 >> v1:172.21.15.190:6800/4220409699 conn(0x7f9a2406ddf0 legacy=0x7f9a240702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:09.038 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.035+0000 7f9a3d881640 10 monclient: shutdown 2024-03-25T18:35:09.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.035+0000 7f9a3d881640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:09.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.035+0000 7f9a3d881640 1 -- 172.21.15.99:0/221043736 >> v1:172.21.15.99:6789/0 conn(0x7f9a381430c0 legacy=0x7f9a381e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:09.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.036+0000 7f9a3d881640 1 -- 172.21.15.99:0/221043736 shutdown_connections 2024-03-25T18:35:09.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.036+0000 7f9a3d881640 1 -- 172.21.15.99:0/221043736 >> 172.21.15.99:0/221043736 conn(0x7f9a380b7ea0 msgr2=0x7f9a38074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:09.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.036+0000 7f9a3d881640 1 -- 172.21.15.99:0/221043736 shutdown_connections 2024-03-25T18:35:09.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.036+0000 7f9a3d881640 1 -- 172.21.15.99:0/221043736 wait complete. 2024-03-25T18:35:09.039 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:09.036+0000 7f9a3d881640 1 librados: shutdown 2024-03-25T18:35:09.049 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:15.191 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:10:15.783 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:15.236 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:10:15.795 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:15.238 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:10:15.797 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:15.245 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:10:15.838 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:15.271 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:10:15.829 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:15.281 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:10:15.873 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:15.285 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:10:15.877 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:15.298 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:10:15.890 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:15.332 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:10:15.925 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:15.856 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:15.969 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.968+0000 7f964a1a3640 10 monclient: get_monmap_and_config 2024-03-25T18:35:15.969 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.968+0000 7f964a1a3640 10 monclient: build_initial_monmap 2024-03-25T18:35:15.969 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.968+0000 7f964a1a3640 10 monclient: monmap: 2024-03-25T18:35:15.969 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:15.969 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:15.970 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:35:15.969697+0000 2024-03-25T18:35:15.970 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:35:15.969697+0000 2024-03-25T18:35:15.970 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:15.970 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:15.970 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:15.970 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:35:15.970 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:35:15.970 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:15.970 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.969+0000 7f964a1a3640 1 Processor -- start 2024-03-25T18:35:15.971 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.969+0000 7f964a1a3640 1 -- start start 2024-03-25T18:35:15.971 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.969+0000 7f964a1a3640 10 monclient: init 2024-03-25T18:35:15.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.970+0000 7f964a1a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:15.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.970+0000 7f964a1a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:15.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.970+0000 7f964a1a3640 10 monclient(hunting): picked mon.noname-c con 0x7f96440b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:15.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.970+0000 7f964a1a3640 10 monclient(hunting): picked mon.noname-a con 0x7f96440b1730 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:15.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.970+0000 7f964a1a3640 10 monclient(hunting): picked mon.noname-b con 0x7f964414be90 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:15.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.970+0000 7f964a1a3640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96440af940 con 0x7f96440b1730 2024-03-25T18:35:15.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.970+0000 7f964a1a3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96440b0a70 con 0x7f96440b4d60 2024-03-25T18:35:15.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.970+0000 7f964a1a3640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96440b0d50 con 0x7f964414be90 2024-03-25T18:35:15.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.970+0000 7f964a1a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:15.972 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.970+0000 7f964a1a3640 10 monclient(hunting): authenticate will time out at 7934.138762s 2024-03-25T18:35:15.978 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.977+0000 7f9642ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f96440b1730 0x7f96440b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37864/0 (socket says 172.21.15.99:37864) 2024-03-25T18:35:15.978 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.977+0000 7f9642ffd640 1 -- 172.21.15.99:0/1444853787 learned_addr learned my addr 172.21.15.99:0/1444853787 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:15.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.993+0000 7f96427fc640 1 -- 172.21.15.99:0/1444853787 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484293733 0 0) 0x7f96440af940 con 0x7f96440b1730 2024-03-25T18:35:15.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.993+0000 7f96427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:15.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.993+0000 7f96427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:15.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.993+0000 7f96427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:15.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.993+0000 7f96427fc640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:35:15.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.994+0000 7f96427fc640 1 -- 172.21.15.99:0/1444853787 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f962c003520 con 0x7f96440b1730 2024-03-25T18:35:15.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.995+0000 7f96427fc640 1 -- 172.21.15.99:0/1444853787 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815027324 0 0) 0x7f96440b0a70 con 0x7f96440b4d60 2024-03-25T18:35:15.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.995+0000 7f96427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:15.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.995+0000 7f96427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:15.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.995+0000 7f96427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:15.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.995+0000 7f96427fc640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:35:15.997 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.995+0000 7f96427fc640 1 -- 172.21.15.99:0/1444853787 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96440af940 con 0x7f96440b4d60 2024-03-25T18:35:16.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.999+0000 7f96427fc640 1 -- 172.21.15.99:0/1444853787 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1164699068 0 0) 0x7f962c003520 con 0x7f96440b1730 2024-03-25T18:35:16.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.999+0000 7f96427fc640 1 -- 172.21.15.99:0/1444853787 >> v1:172.21.15.99:6790/0 conn(0x7f964414be90 legacy=0x7f964414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:16.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.999+0000 7f96427fc640 1 -- 172.21.15.99:0/1444853787 >> v1:172.21.15.190:6789/0 conn(0x7f96440b4d60 legacy=0x7f964413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:16.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.999+0000 7f96427fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:16.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.999+0000 7f96427fc640 1 monclient: found mon.noname-a 2024-03-25T18:35:16.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:15.999+0000 7f96427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:16.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.000+0000 7f96427fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:35:16.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.000+0000 7f96427fc640 1 -- 172.21.15.99:0/1444853787 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96440b1030 con 0x7f96440b1730 2024-03-25T18:35:16.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.000+0000 7f96427fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:16.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.000+0000 7f96427fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:16.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.000+0000 7f96427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:16.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.000+0000 7f96427fc640 1 -- 172.21.15.99:0/1444853787 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9638001690 con 0x7f96440b1730 2024-03-25T18:35:16.001 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.000+0000 7f96427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:16.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.000+0000 7f96427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:16.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.000+0000 7f96427fc640 10 monclient: dump: 2024-03-25T18:35:16.002 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:16.002 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:16.002 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:16.002 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:16.002 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:16.002 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:16.002 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:16.002 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:16.002 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:16.002 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:16.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.000+0000 7f964a1a3640 5 monclient: authenticate success, global_id 4662 2024-03-25T18:35:16.002 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.000+0000 7f964a1a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:16.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.015+0000 7f96427fc640 1 -- 172.21.15.99:0/1444853787 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96380017f0 con 0x7f96440b1730 2024-03-25T18:35:16.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.015+0000 7f96427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:16.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.015+0000 7f96427fc640 1 -- 172.21.15.99:0/1444853787 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9638001c00 con 0x7f96440b1730 2024-03-25T18:35:16.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.015+0000 7f964a1a3640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:16.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.015+0000 7f96427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:16.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.015+0000 7f964a1a3640 4 set_mon_vals no callback set 2024-03-25T18:35:16.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.015+0000 7f96427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:16.016 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.015+0000 7f96427fc640 10 monclient: dump: 2024-03-25T18:35:16.016 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:16.016 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:16.017 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:16.017 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:16.017 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:16.017 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:16.017 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:16.017 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:16.017 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:16.017 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:16.019 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.016+0000 7f964a1a3640 10 monclient: shutdown 2024-03-25T18:35:16.019 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.016+0000 7f964a1a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:16.019 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.016+0000 7f964a1a3640 1 -- 172.21.15.99:0/1444853787 >> v1:172.21.15.99:6789/0 conn(0x7f96440b1730 legacy=0x7f96440b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:16.019 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.016+0000 7f964a1a3640 1 -- 172.21.15.99:0/1444853787 shutdown_connections 2024-03-25T18:35:16.019 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.016+0000 7f964a1a3640 1 -- 172.21.15.99:0/1444853787 >> 172.21.15.99:0/1444853787 conn(0x7f96440617a0 msgr2=0x7f9644061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:16.019 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.016+0000 7f964a1a3640 1 -- 172.21.15.99:0/1444853787 shutdown_connections 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.016+0000 7f964a1a3640 1 -- 172.21.15.99:0/1444853787 wait complete. 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.017+0000 7f964a1a3640 10 monclient: build_initial_monmap 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.017+0000 7f964a1a3640 10 monclient: monmap: 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.017+0000 7f964a1a3640 1 librados: starting msgr at 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.017+0000 7f964a1a3640 1 librados: starting objecter 2024-03-25T18:35:16.020 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.017+0000 7f964a1a3640 1 Processor -- start 2024-03-25T18:35:16.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.017+0000 7f964a1a3640 1 -- start start 2024-03-25T18:35:16.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.017+0000 7f964a1a3640 1 librados: setting wanted keys 2024-03-25T18:35:16.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.017+0000 7f964a1a3640 1 librados: calling monclient init 2024-03-25T18:35:16.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.017+0000 7f964a1a3640 10 monclient: init 2024-03-25T18:35:16.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.018+0000 7f964a1a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:16.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.018+0000 7f964a1a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:16.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.018+0000 7f964a1a3640 10 monclient(hunting): picked mon.noname-c con 0x7f96440b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:16.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.018+0000 7f964a1a3640 10 monclient(hunting): picked mon.noname-b con 0x7f96440b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:16.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.018+0000 7f964a1a3640 10 monclient(hunting): picked mon.noname-a con 0x7f964414be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:16.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.018+0000 7f964a1a3640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964412d790 con 0x7f964414be90 2024-03-25T18:35:16.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.018+0000 7f964a1a3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9644136770 con 0x7f96440b4d60 2024-03-25T18:35:16.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.018+0000 7f964a1a3640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9644136a50 con 0x7f96440b1730 2024-03-25T18:35:16.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.018+0000 7f964a1a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:16.021 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.018+0000 7f964a1a3640 10 monclient(hunting): authenticate will time out at 7934.186244s 2024-03-25T18:35:16.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.027+0000 7f96437fe640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f96440b1730 0x7f9644137530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:41000/0 (socket says 172.21.15.99:41000) 2024-03-25T18:35:16.028 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.027+0000 7f96437fe640 1 -- 172.21.15.99:0/3591396680 learned_addr learned my addr 172.21.15.99:0/3591396680 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:16.029 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.028+0000 7f962bfff640 10 client.?.objecter ms_handle_connect 0x7f96440b1730 2024-03-25T18:35:16.029 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.028+0000 7f962bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:16.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.030+0000 7f962bfff640 10 client.?.objecter ms_handle_connect 0x7f964414be90 2024-03-25T18:35:16.031 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.030+0000 7f962bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:16.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.034+0000 7f962bfff640 10 client.?.objecter ms_handle_connect 0x7f96440b4d60 2024-03-25T18:35:16.035 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.034+0000 7f962bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:16.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.039+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988334153 0 0) 0x7f9644136770 con 0x7f96440b4d60 2024-03-25T18:35:16.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.039+0000 7f962bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:16.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.039+0000 7f962bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:16.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.039+0000 7f962bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:16.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.039+0000 7f962bfff640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:35:16.040 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.039+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9630004eb0 con 0x7f96440b4d60 2024-03-25T18:35:16.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.040+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448675485 0 0) 0x7f9644136a50 con 0x7f96440b1730 2024-03-25T18:35:16.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.040+0000 7f962bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:16.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.040+0000 7f962bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:16.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.040+0000 7f962bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:16.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.040+0000 7f962bfff640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:35:16.041 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.040+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9644136770 con 0x7f96440b1730 2024-03-25T18:35:16.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.048+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921706457 0 0) 0x7f964412d790 con 0x7f964414be90 2024-03-25T18:35:16.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.048+0000 7f962bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:35:16.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.048+0000 7f962bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:16.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.048+0000 7f962bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:16.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.048+0000 7f962bfff640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:35:16.049 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.048+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9644136a50 con 0x7f964414be90 2024-03-25T18:35:16.056 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.055+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 797425206 0 0) 0x7f9644136770 con 0x7f96440b1730 2024-03-25T18:35:16.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.055+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f964412d790 con 0x7f96440b1730 2024-03-25T18:35:16.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.055+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f96340019f0 con 0x7f96440b1730 2024-03-25T18:35:16.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.055+0000 7f962bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:16.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.055+0000 7f962bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:16.057 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.055+0000 7f962bfff640 10 monclient(hunting): dump: 2024-03-25T18:35:16.057 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:16.057 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:16.057 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:16.057 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:16.057 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:16.057 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:16.057 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:16.057 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:16.057 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:16.058 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:16.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.055+0000 7f962bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:16.058 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.057+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1041338274 0 0) 0x7f9630004eb0 con 0x7f96440b4d60 2024-03-25T18:35:16.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.057+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9644136770 con 0x7f96440b4d60 2024-03-25T18:35:16.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.057+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9638001bc0 con 0x7f96440b4d60 2024-03-25T18:35:16.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.057+0000 7f962bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:16.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.057+0000 7f962bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:16.059 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.057+0000 7f962bfff640 10 monclient(hunting): dump: 2024-03-25T18:35:16.059 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:16.059 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:16.059 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:16.059 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:16.059 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:16.059 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:16.059 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:16.059 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:16.059 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:16.060 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:16.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.062+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3122552891 0 0) 0x7f9644136a50 con 0x7f964414be90 2024-03-25T18:35:16.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.062+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9630004eb0 con 0x7f964414be90 2024-03-25T18:35:16.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.062+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f963c001a60 con 0x7f964414be90 2024-03-25T18:35:16.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.062+0000 7f962bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:16.063 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.062+0000 7f962bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:16.064 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.062+0000 7f962bfff640 10 monclient(hunting): dump: 2024-03-25T18:35:16.064 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:16.064 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:16.064 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:16.064 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:16.064 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:16.064 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:16.064 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:16.064 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:16.064 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:16.064 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:16.066 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.064+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3393160015 0 0) 0x7f9644136770 con 0x7f96440b4d60 2024-03-25T18:35:16.066 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.064+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 >> v1:172.21.15.99:6790/0 conn(0x7f96440b1730 legacy=0x7f9644137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:16.066 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.064+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 >> v1:172.21.15.99:6789/0 conn(0x7f964414be90 legacy=0x7f964412d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:16.066 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.064+0000 7f962bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:35:16.066 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.064+0000 7f962bfff640 1 monclient: found mon.b 2024-03-25T18:35:16.066 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f962bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:16.066 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f962bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:16.066 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9644136d30 con 0x7f96440b4d60 2024-03-25T18:35:16.066 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f962bfff640 10 monclient: _finish_auth 0 2024-03-25T18:35:16.066 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f962bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:35:16.066 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f962bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:16.066 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f964a1a3640 5 monclient: authenticate success, global_id 4675 2024-03-25T18:35:16.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f964a1a3640 10 monclient: _renew_subs 2024-03-25T18:35:16.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f964a1a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:16.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f964a1a3640 1 -- 172.21.15.99:0/3591396680 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964413e370 con 0x7f96440b4d60 2024-03-25T18:35:16.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f964a1a3640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:16.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f964a1a3640 10 monclient: _renew_subs 2024-03-25T18:35:16.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f964a1a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:16.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f964a1a3640 1 -- 172.21.15.99:0/3591396680 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f964413dfa0 con 0x7f96440b4d60 2024-03-25T18:35:16.067 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.065+0000 7f964a1a3640 1 librados: init done 2024-03-25T18:35:16.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.066+0000 7f964a1a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:16.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.066+0000 7f964a1a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:16.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.066+0000 7f964a1a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:16.068 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.066+0000 7f964a1a3640 1 -- 172.21.15.99:0/3591396680 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96441401f0 con 0x7f96440b4d60 2024-03-25T18:35:16.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.337+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9638004a90 con 0x7f96440b4d60 2024-03-25T18:35:16.338 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.337+0000 7f962bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.337+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9638004da0 con 0x7f96440b4d60 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.337+0000 7f962bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.337+0000 7f962bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.337+0000 7f962bfff640 10 monclient: dump: 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:16.339 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.337+0000 7f9641ffb640 4 set_mon_vals no callback set 2024-03-25T18:35:16.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.338+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f963801ad70 con 0x7f96440b4d60 2024-03-25T18:35:16.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.339+0000 7f96437fe640 10 monclient: get_auth_request con 0x7f963006e160 auth_method 0 2024-03-25T18:35:16.340 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.339+0000 7f962bfff640 10 client.4675.objecter ms_handle_connect 0x7f963006e160 2024-03-25T18:35:16.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.344+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 114198336 0 0) 0x7f9638054930 con 0x7f96440b4d60 2024-03-25T18:35:16.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.344+0000 7f962bfff640 10 client.4675.objecter ms_dispatch 0x7f9644000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:16.345 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.344+0000 7f962bfff640 3 client.4675.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:16.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.344+0000 7f962bfff640 3 client.4675.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:16.346 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.345+0000 7f962bfff640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T18:35:16.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.347+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9638050270 con 0x7f96440b4d60 2024-03-25T18:35:16.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.347+0000 7f962bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:16.348 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.347+0000 7f962bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:16.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.568+0000 7f964a1a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:16.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.568+0000 7f964a1a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:16.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.568+0000 7f964a1a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:16.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.568+0000 7f964a1a3640 1 -- 172.21.15.99:0/3591396680 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96440afbd0 con 0x7f96440b4d60 2024-03-25T18:35:16.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.614+0000 7f962bfff640 1 -- 172.21.15.99:0/3591396680 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 727448183) 0x7f9638052370 con 0x7f96440b4d60 2024-03-25T18:35:16.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.614+0000 7f962bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:16.616 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.614+0000 7f962bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:16.616 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:35:16.616 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:25:21.670672+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:25:21.670672+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:16.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.617+0000 7f964a1a3640 10 librados: watch_flush enter 2024-03-25T18:35:16.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.617+0000 7f964a1a3640 10 librados: watch_flush exit 2024-03-25T18:35:16.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.617+0000 7f964a1a3640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T18:35:16.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.617+0000 7f964a1a3640 10 client.4675.objecter successfully canceled tick 2024-03-25T18:35:16.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.617+0000 7f964a1a3640 1 -- 172.21.15.99:0/3591396680 >> v1:172.21.15.190:6800/4220409699 conn(0x7f963006e160 legacy=0x7f9630070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:16.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.617+0000 7f964a1a3640 10 monclient: shutdown 2024-03-25T18:35:16.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.617+0000 7f964a1a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:16.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.617+0000 7f964a1a3640 1 -- 172.21.15.99:0/3591396680 >> v1:172.21.15.190:6789/0 conn(0x7f96440b4d60 legacy=0x7f964412c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:16.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.617+0000 7f964a1a3640 1 -- 172.21.15.99:0/3591396680 shutdown_connections 2024-03-25T18:35:16.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.617+0000 7f964a1a3640 1 -- 172.21.15.99:0/3591396680 >> 172.21.15.99:0/3591396680 conn(0x7f96440617a0 msgr2=0x7f964413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:16.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.617+0000 7f964a1a3640 1 -- 172.21.15.99:0/3591396680 shutdown_connections 2024-03-25T18:35:16.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.618+0000 7f964a1a3640 1 -- 172.21.15.99:0/3591396680 wait complete. 2024-03-25T18:35:16.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:16.618+0000 7f964a1a3640 1 librados: shutdown 2024-03-25T18:35:16.629 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:23.436 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:23.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.551+0000 7fd308496640 10 monclient: get_monmap_and_config 2024-03-25T18:35:23.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.551+0000 7fd308496640 10 monclient: build_initial_monmap 2024-03-25T18:35:23.553 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.552+0000 7fd308496640 10 monclient: monmap: 2024-03-25T18:35:23.553 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:23.553 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:23.553 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:35:23.553031+0000 2024-03-25T18:35:23.553 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:35:23.553031+0000 2024-03-25T18:35:23.553 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:23.553 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:23.553 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:23.553 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:35:23.553 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:35:23.554 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:23.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.552+0000 7fd308496640 1 Processor -- start 2024-03-25T18:35:23.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.552+0000 7fd308496640 1 -- start start 2024-03-25T18:35:23.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.552+0000 7fd308496640 10 monclient: init 2024-03-25T18:35:23.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.553+0000 7fd308496640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:23.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.553+0000 7fd308496640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:23.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.553+0000 7fd308496640 10 monclient(hunting): picked mon.noname-b con 0x7fd3000b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:23.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.553+0000 7fd308496640 10 monclient(hunting): picked mon.noname-c con 0x7fd3000b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:23.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.553+0000 7fd308496640 10 monclient(hunting): picked mon.noname-a con 0x7fd30014be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:23.555 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.553+0000 7fd308496640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3000af940 con 0x7fd30014be90 2024-03-25T18:35:23.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.553+0000 7fd308496640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3000b0a70 con 0x7fd3000b1730 2024-03-25T18:35:23.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.553+0000 7fd308496640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3000b0d50 con 0x7fd3000b4d60 2024-03-25T18:35:23.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.553+0000 7fd308496640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:23.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.553+0000 7fd308496640 10 monclient(hunting): authenticate will time out at 7941.721805s 2024-03-25T18:35:23.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.562+0000 7fd305a31640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fd3000b1730 0x7fd3000b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:43448/0 (socket says 172.21.15.99:43448) 2024-03-25T18:35:23.563 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.562+0000 7fd305a31640 1 -- 172.21.15.99:0/3889006521 learned_addr learned my addr 172.21.15.99:0/3889006521 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:23.575 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.574+0000 7fd305230640 1 -- 172.21.15.99:0/3889006521 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539795431 0 0) 0x7fd3000b0a70 con 0x7fd3000b1730 2024-03-25T18:35:23.575 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.574+0000 7fd305230640 10 monclient(hunting): _negotiate 2024-03-25T18:35:23.575 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.574+0000 7fd305230640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:23.576 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.574+0000 7fd305230640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:23.576 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.574+0000 7fd305230640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:35:23.576 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.574+0000 7fd305230640 1 -- 172.21.15.99:0/3889006521 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ec003520 con 0x7fd3000b1730 2024-03-25T18:35:23.581 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.580+0000 7fd305230640 1 -- 172.21.15.99:0/3889006521 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599052188 0 0) 0x7fd3000b0d50 con 0x7fd3000b4d60 2024-03-25T18:35:23.581 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.580+0000 7fd305230640 10 monclient(hunting): _negotiate 2024-03-25T18:35:23.581 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.580+0000 7fd305230640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:23.581 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.580+0000 7fd305230640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:23.581 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.580+0000 7fd305230640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:35:23.581 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.580+0000 7fd305230640 1 -- 172.21.15.99:0/3889006521 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3000b0a70 con 0x7fd3000b4d60 2024-03-25T18:35:23.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 1 -- 172.21.15.99:0/3889006521 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 410242840 0 0) 0x7fd2ec003520 con 0x7fd3000b1730 2024-03-25T18:35:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 1 -- 172.21.15.99:0/3889006521 >> v1:172.21.15.99:6790/0 conn(0x7fd3000b4d60 legacy=0x7fd30013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 1 -- 172.21.15.99:0/3889006521 >> v1:172.21.15.99:6789/0 conn(0x7fd30014be90 legacy=0x7fd30014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:35:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 10 monclient: _finish_hunting 0 2024-03-25T18:35:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 1 monclient: found mon.noname-c 2024-03-25T18:35:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:35:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 1 -- 172.21.15.99:0/3889006521 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3000b1030 con 0x7fd3000b1730 2024-03-25T18:35:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 10 monclient: _finish_auth 0 2024-03-25T18:35:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 10 monclient: _check_auth_tickets 2024-03-25T18:35:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 1 -- 172.21.15.99:0/3889006521 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fd2f0001690 con 0x7fd3000b1730 2024-03-25T18:35:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:23.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd305230640 10 monclient: dump: 2024-03-25T18:35:23.585 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:23.585 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:23.585 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:23.585 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:23.585 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:23.585 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:23.585 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:23.585 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:23.585 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:23.585 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:23.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd308496640 5 monclient: authenticate success, global_id 4678 2024-03-25T18:35:23.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.582+0000 7fd308496640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:23.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.596+0000 7fd305230640 1 -- 172.21.15.99:0/3889006521 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2f00017f0 con 0x7fd3000b1730 2024-03-25T18:35:23.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.596+0000 7fd305230640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:23.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.596+0000 7fd305230640 1 -- 172.21.15.99:0/3889006521 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fd2f0001c00 con 0x7fd3000b1730 2024-03-25T18:35:23.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.596+0000 7fd308496640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:23.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.596+0000 7fd308496640 4 set_mon_vals no callback set 2024-03-25T18:35:23.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.596+0000 7fd305230640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:23.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.596+0000 7fd305230640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:23.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.596+0000 7fd305230640 10 monclient: dump: 2024-03-25T18:35:23.597 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:23.597 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:23.597 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:23.597 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:23.598 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:23.598 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:23.598 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:23.598 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:23.598 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:23.598 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:23.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.596+0000 7fd308496640 10 monclient: shutdown 2024-03-25T18:35:23.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.596+0000 7fd308496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:23.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.596+0000 7fd308496640 1 -- 172.21.15.99:0/3889006521 >> v1:172.21.15.190:6789/0 conn(0x7fd3000b1730 legacy=0x7fd3000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.596+0000 7fd308496640 1 -- 172.21.15.99:0/3889006521 shutdown_connections 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.596+0000 7fd308496640 1 -- 172.21.15.99:0/3889006521 >> 172.21.15.99:0/3889006521 conn(0x7fd3000617a0 msgr2=0x7fd300061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.596+0000 7fd308496640 1 -- 172.21.15.99:0/3889006521 shutdown_connections 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.597+0000 7fd308496640 1 -- 172.21.15.99:0/3889006521 wait complete. 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.597+0000 7fd308496640 10 monclient: build_initial_monmap 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.597+0000 7fd308496640 10 monclient: monmap: 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:35:23.600 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:35:23.601 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:23.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.598+0000 7fd308496640 1 librados: starting msgr at 2024-03-25T18:35:23.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.598+0000 7fd308496640 1 librados: starting objecter 2024-03-25T18:35:23.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.598+0000 7fd308496640 1 Processor -- start 2024-03-25T18:35:23.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.598+0000 7fd308496640 1 -- start start 2024-03-25T18:35:23.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.598+0000 7fd308496640 1 librados: setting wanted keys 2024-03-25T18:35:23.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.598+0000 7fd308496640 1 librados: calling monclient init 2024-03-25T18:35:23.601 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.598+0000 7fd308496640 10 monclient: init 2024-03-25T18:35:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.599+0000 7fd308496640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.599+0000 7fd308496640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.599+0000 7fd308496640 10 monclient(hunting): picked mon.noname-c con 0x7fd3000b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.599+0000 7fd308496640 10 monclient(hunting): picked mon.noname-a con 0x7fd3000b4d60 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.599+0000 7fd308496640 10 monclient(hunting): picked mon.noname-b con 0x7fd30014be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.599+0000 7fd308496640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd300139770 con 0x7fd3000b4d60 2024-03-25T18:35:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.599+0000 7fd308496640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3001dbc30 con 0x7fd30014be90 2024-03-25T18:35:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.599+0000 7fd308496640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd300138dd0 con 0x7fd3000b1730 2024-03-25T18:35:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.599+0000 7fd308496640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:23.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.599+0000 7fd308496640 10 monclient(hunting): authenticate will time out at 7941.767264s 2024-03-25T18:35:23.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.603+0000 7fd306a33640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fd30014be90 0x7fd3001e2140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:43496/0 (socket says 172.21.15.99:43496) 2024-03-25T18:35:23.604 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.603+0000 7fd306a33640 1 -- 172.21.15.99:0/2687632455 learned_addr learned my addr 172.21.15.99:0/2687632455 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:23.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.604+0000 7fd2f6ffd640 10 client.?.objecter ms_handle_connect 0x7fd30014be90 2024-03-25T18:35:23.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.604+0000 7fd2f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:23.608 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.607+0000 7fd2f6ffd640 10 client.?.objecter ms_handle_connect 0x7fd3000b1730 2024-03-25T18:35:23.608 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.607+0000 7fd2f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:23.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.613+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035993974 0 0) 0x7fd300138dd0 con 0x7fd3000b1730 2024-03-25T18:35:23.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.613+0000 7fd2f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:23.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.613+0000 7fd2f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:23.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.613+0000 7fd2f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:23.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.613+0000 7fd2f6ffd640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:35:23.614 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.613+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e4004eb0 con 0x7fd3000b1730 2024-03-25T18:35:23.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.614+0000 7fd2f6ffd640 10 client.?.objecter ms_handle_connect 0x7fd3000b4d60 2024-03-25T18:35:23.615 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.614+0000 7fd2f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:23.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.618+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242148500 0 0) 0x7fd3001dbc30 con 0x7fd30014be90 2024-03-25T18:35:23.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.618+0000 7fd2f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:23.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.618+0000 7fd2f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:23.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.618+0000 7fd2f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:23.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.618+0000 7fd2f6ffd640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:35:23.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.618+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd300138dd0 con 0x7fd30014be90 2024-03-25T18:35:23.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.622+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4266917313 0 0) 0x7fd2e4004eb0 con 0x7fd3000b1730 2024-03-25T18:35:23.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.622+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3001dbc30 con 0x7fd3000b1730 2024-03-25T18:35:23.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.622+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fd2fc001930 con 0x7fd3000b1730 2024-03-25T18:35:23.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.622+0000 7fd2f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:23.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.622+0000 7fd2f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:23.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.622+0000 7fd2f6ffd640 10 monclient(hunting): dump: 2024-03-25T18:35:23.624 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:23.624 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:23.624 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:23.624 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:23.624 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:23.624 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:23.624 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:23.624 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:23.624 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:23.624 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:23.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.622+0000 7fd2f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:23.634 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.633+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873891742 0 0) 0x7fd300139770 con 0x7fd3000b4d60 2024-03-25T18:35:23.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.633+0000 7fd2f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:23.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.633+0000 7fd2f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:23.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.633+0000 7fd2f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:23.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.633+0000 7fd2f6ffd640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:35:23.635 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.633+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e4004eb0 con 0x7fd3000b4d60 2024-03-25T18:35:23.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2421327614 0 0) 0x7fd3001dbc30 con 0x7fd3000b1730 2024-03-25T18:35:23.643 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 >> v1:172.21.15.190:6789/0 conn(0x7fd30014be90 legacy=0x7fd3001e2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:23.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 >> v1:172.21.15.99:6789/0 conn(0x7fd3000b4d60 legacy=0x7fd3001db520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:23.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd2f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:23.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd2f6ffd640 1 monclient: found mon.c 2024-03-25T18:35:23.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd2f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:23.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd2f6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:35:23.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd300138f60 con 0x7fd3000b1730 2024-03-25T18:35:23.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd2f6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:35:23.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd2f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:23.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd2f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:23.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd308496640 5 monclient: authenticate success, global_id 4676 2024-03-25T18:35:23.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd308496640 10 monclient: _renew_subs 2024-03-25T18:35:23.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd308496640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:35:23.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd308496640 1 -- 172.21.15.99:0/2687632455 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd300135920 con 0x7fd3000b1730 2024-03-25T18:35:23.644 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd308496640 10 client.4676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:23.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd308496640 10 monclient: _renew_subs 2024-03-25T18:35:23.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd308496640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:35:23.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd308496640 1 -- 172.21.15.99:0/2687632455 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd300135640 con 0x7fd3000b1730 2024-03-25T18:35:23.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.642+0000 7fd308496640 1 librados: init done 2024-03-25T18:35:23.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.643+0000 7fd308496640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:23.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.643+0000 7fd308496640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:23.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.643+0000 7fd308496640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:35:23.645 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.643+0000 7fd308496640 1 -- 172.21.15.99:0/2687632455 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3000627b0 con 0x7fd3000b1730 2024-03-25T18:35:23.992 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.991+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2fc003610 con 0x7fd3000b1730 2024-03-25T18:35:23.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.991+0000 7fd2f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:23.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.991+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fd2fc003920 con 0x7fd3000b1730 2024-03-25T18:35:23.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.991+0000 7fd2f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:23.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.991+0000 7fd2f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:23.993 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.991+0000 7fd2f6ffd640 10 monclient: dump: 2024-03-25T18:35:23.993 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:23.993 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:23.993 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:23.993 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:23.993 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:23.993 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:23.993 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:23.993 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:23.994 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:23.994 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:23.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.991+0000 7fd304a2f640 4 set_mon_vals no callback set 2024-03-25T18:35:23.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.992+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fd2fc0198f0 con 0x7fd3000b1730 2024-03-25T18:35:23.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.993+0000 7fd305a31640 10 monclient: get_auth_request con 0x7fd2e406ddb0 auth_method 0 2024-03-25T18:35:23.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.993+0000 7fd2f6ffd640 10 client.4676.objecter ms_handle_connect 0x7fd2e406ddb0 2024-03-25T18:35:23.999 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.998+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2833420883 0 0) 0x7fd2fc053670 con 0x7fd3000b1730 2024-03-25T18:35:23.999 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.998+0000 7fd2f6ffd640 10 client.4676.objecter ms_dispatch 0x7fd300000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:23.999 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.998+0000 7fd2f6ffd640 3 client.4676.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:24.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.998+0000 7fd2f6ffd640 3 client.4676.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:24.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.998+0000 7fd2f6ffd640 20 client.4676.objecter dump_active .. 0 homeless 2024-03-25T18:35:24.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.998+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2fc0b08b0 con 0x7fd3000b1730 2024-03-25T18:35:24.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.998+0000 7fd2f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:24.000 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:23.998+0000 7fd2f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:24.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.222+0000 7fd308496640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:24.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.222+0000 7fd308496640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:24.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.222+0000 7fd308496640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:35:24.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.222+0000 7fd308496640 1 -- 172.21.15.99:0/2687632455 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd300145010 con 0x7fd3000b1730 2024-03-25T18:35:24.252 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.251+0000 7fd2f6ffd640 1 -- 172.21.15.99:0/2687632455 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 270404165) 0x7fd2fc04efb0 con 0x7fd3000b1730 2024-03-25T18:35:24.252 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.251+0000 7fd2f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:24.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.251+0000 7fd2f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:24.253 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:35:24.253 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":11,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:35:20.161948+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:20.161948+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:24.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.253+0000 7fd308496640 10 librados: watch_flush enter 2024-03-25T18:35:24.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.254+0000 7fd308496640 10 librados: watch_flush exit 2024-03-25T18:35:24.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.254+0000 7fd308496640 20 client.4676.objecter shutdown clearing up homeless session... 2024-03-25T18:35:24.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.254+0000 7fd308496640 10 client.4676.objecter successfully canceled tick 2024-03-25T18:35:24.255 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.254+0000 7fd308496640 1 -- 172.21.15.99:0/2687632455 >> v1:172.21.15.190:6800/4220409699 conn(0x7fd2e406ddb0 legacy=0x7fd2e4070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:24.256 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.254+0000 7fd308496640 10 monclient: shutdown 2024-03-25T18:35:24.256 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.254+0000 7fd308496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:24.256 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.254+0000 7fd308496640 1 -- 172.21.15.99:0/2687632455 >> v1:172.21.15.99:6790/0 conn(0x7fd3000b1730 legacy=0x7fd3001dae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:24.256 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.254+0000 7fd308496640 1 -- 172.21.15.99:0/2687632455 shutdown_connections 2024-03-25T18:35:24.256 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.254+0000 7fd308496640 1 -- 172.21.15.99:0/2687632455 >> 172.21.15.99:0/2687632455 conn(0x7fd3000617a0 msgr2=0x7fd30013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:24.256 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.254+0000 7fd308496640 1 -- 172.21.15.99:0/2687632455 shutdown_connections 2024-03-25T18:35:24.256 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.254+0000 7fd308496640 1 -- 172.21.15.99:0/2687632455 wait complete. 2024-03-25T18:35:24.256 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:24.254+0000 7fd308496640 1 librados: shutdown 2024-03-25T18:35:24.266 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:31.073 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:31.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.164+0000 7f0e14df6640 10 monclient: get_monmap_and_config 2024-03-25T18:35:31.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.164+0000 7f0e14df6640 10 monclient: build_initial_monmap 2024-03-25T18:35:31.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.164+0000 7f0e14df6640 10 monclient: monmap: 2024-03-25T18:35:31.166 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:31.166 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:31.166 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:35:31.165865+0000 2024-03-25T18:35:31.166 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:35:31.165865+0000 2024-03-25T18:35:31.166 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:31.166 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:31.166 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:31.166 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:35:31.166 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:35:31.166 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:31.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.165+0000 7f0e14df6640 1 Processor -- start 2024-03-25T18:35:31.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.165+0000 7f0e14df6640 1 -- start start 2024-03-25T18:35:31.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.165+0000 7f0e14df6640 10 monclient: init 2024-03-25T18:35:31.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.166+0000 7f0e14df6640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:31.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.166+0000 7f0e14df6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:31.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.166+0000 7f0e14df6640 10 monclient(hunting): picked mon.noname-c con 0x7f0e100b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:31.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.166+0000 7f0e14df6640 10 monclient(hunting): picked mon.noname-b con 0x7f0e100b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:31.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.166+0000 7f0e14df6640 10 monclient(hunting): picked mon.noname-a con 0x7f0e1014be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:31.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.166+0000 7f0e14df6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e100af940 con 0x7f0e1014be90 2024-03-25T18:35:31.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.166+0000 7f0e14df6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e100b0a70 con 0x7f0e100b4d60 2024-03-25T18:35:31.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.166+0000 7f0e14df6640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e100b0d50 con 0x7f0e100b1730 2024-03-25T18:35:31.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.166+0000 7f0e14df6640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:31.168 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.166+0000 7f0e14df6640 10 monclient(hunting): authenticate will time out at 7949.334411s 2024-03-25T18:35:31.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.175+0000 7f0e0dd9b640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f0e100b1730 0x7f0e100b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:44742/0 (socket says 172.21.15.99:44742) 2024-03-25T18:35:31.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.175+0000 7f0e0dd9b640 1 -- 172.21.15.99:0/2779388772 learned_addr learned my addr 172.21.15.99:0/2779388772 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:31.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.185+0000 7f0e0d59a640 1 -- 172.21.15.99:0/2779388772 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127313583 0 0) 0x7f0e100b0d50 con 0x7f0e100b1730 2024-03-25T18:35:31.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.185+0000 7f0e0d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:31.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.185+0000 7f0e0d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:31.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.185+0000 7f0e0d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:31.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.185+0000 7f0e0d59a640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:35:31.186 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.185+0000 7f0e0d59a640 1 -- 172.21.15.99:0/2779388772 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0df8003520 con 0x7f0e100b1730 2024-03-25T18:35:31.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 1 -- 172.21.15.99:0/2779388772 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2906949591 0 0) 0x7f0df8003520 con 0x7f0e100b1730 2024-03-25T18:35:31.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 1 -- 172.21.15.99:0/2779388772 >> v1:172.21.15.190:6789/0 conn(0x7f0e100b4d60 legacy=0x7f0e1013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:35:31.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 1 -- 172.21.15.99:0/2779388772 >> v1:172.21.15.99:6789/0 conn(0x7f0e1014be90 legacy=0x7f0e1014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:31.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:35:31.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 1 monclient: found mon.noname-b 2024-03-25T18:35:31.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:31.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:35:31.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 1 -- 172.21.15.99:0/2779388772 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e100b1030 con 0x7f0e100b1730 2024-03-25T18:35:31.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 10 monclient: _finish_auth 0 2024-03-25T18:35:31.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:35:31.194 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 1 -- 172.21.15.99:0/2779388772 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0e04001690 con 0x7f0e100b1730 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e0d59a640 10 monclient: dump: 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e14df6640 5 monclient: authenticate success, global_id 4679 2024-03-25T18:35:31.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.192+0000 7f0e14df6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:31.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.212+0000 7f0e0d59a640 1 -- 172.21.15.99:0/2779388772 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e04001860 con 0x7f0e100b1730 2024-03-25T18:35:31.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.212+0000 7f0e0d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:31.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.212+0000 7f0e0d59a640 1 -- 172.21.15.99:0/2779388772 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0e04001c70 con 0x7f0e100b1730 2024-03-25T18:35:31.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.212+0000 7f0e0d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:31.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.212+0000 7f0e0d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:31.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.212+0000 7f0e0d59a640 10 monclient: dump: 2024-03-25T18:35:31.213 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:31.213 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:31.213 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:31.213 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:31.213 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:31.213 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:31.213 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:31.214 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:31.214 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:31.214 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:31.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.212+0000 7f0e14df6640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:31.214 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.212+0000 7f0e14df6640 4 set_mon_vals no callback set 2024-03-25T18:35:31.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.213+0000 7f0e14df6640 10 monclient: shutdown 2024-03-25T18:35:31.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.213+0000 7f0e14df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:31.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.213+0000 7f0e14df6640 1 -- 172.21.15.99:0/2779388772 >> v1:172.21.15.99:6790/0 conn(0x7f0e100b1730 legacy=0x7f0e100b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:31.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.213+0000 7f0e14df6640 1 -- 172.21.15.99:0/2779388772 shutdown_connections 2024-03-25T18:35:31.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.213+0000 7f0e14df6640 1 -- 172.21.15.99:0/2779388772 >> 172.21.15.99:0/2779388772 conn(0x7f0e100617a0 msgr2=0x7f0e10061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:31.215 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.213+0000 7f0e14df6640 1 -- 172.21.15.99:0/2779388772 shutdown_connections 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.213+0000 7f0e14df6640 1 -- 172.21.15.99:0/2779388772 wait complete. 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.214+0000 7f0e14df6640 10 monclient: build_initial_monmap 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.214+0000 7f0e14df6640 10 monclient: monmap: 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.214+0000 7f0e14df6640 1 librados: starting msgr at 2024-03-25T18:35:31.216 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.214+0000 7f0e14df6640 1 librados: starting objecter 2024-03-25T18:35:31.217 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.214+0000 7f0e14df6640 1 Processor -- start 2024-03-25T18:35:31.217 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.214+0000 7f0e14df6640 1 -- start start 2024-03-25T18:35:31.217 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.214+0000 7f0e14df6640 1 librados: setting wanted keys 2024-03-25T18:35:31.217 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.214+0000 7f0e14df6640 1 librados: calling monclient init 2024-03-25T18:35:31.217 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.214+0000 7f0e14df6640 10 monclient: init 2024-03-25T18:35:31.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.215+0000 7f0e14df6640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:31.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.215+0000 7f0e14df6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:31.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.215+0000 7f0e14df6640 10 monclient(hunting): picked mon.noname-c con 0x7f0e100b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:31.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.215+0000 7f0e14df6640 10 monclient(hunting): picked mon.noname-b con 0x7f0e100b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:31.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.215+0000 7f0e14df6640 10 monclient(hunting): picked mon.noname-a con 0x7f0e1014be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:31.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.215+0000 7f0e14df6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e10143de0 con 0x7f0e1014be90 2024-03-25T18:35:31.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.215+0000 7f0e14df6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e101454b0 con 0x7f0e100b4d60 2024-03-25T18:35:31.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.215+0000 7f0e14df6640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e101442a0 con 0x7f0e100b1730 2024-03-25T18:35:31.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.215+0000 7f0e14df6640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:31.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.215+0000 7f0e14df6640 10 monclient(hunting): authenticate will time out at 7949.383205s 2024-03-25T18:35:31.221 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.221+0000 7f0e0dd9b640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f0e100b4d60 0x7f0e1012fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:60430/0 (socket says 172.21.15.99:60430) 2024-03-25T18:35:31.221 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.221+0000 7f0e0dd9b640 1 -- 172.21.15.99:0/2707245030 learned_addr learned my addr 172.21.15.99:0/2707245030 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:31.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.221+0000 7f0dfeffd640 10 client.?.objecter ms_handle_connect 0x7f0e100b4d60 2024-03-25T18:35:31.222 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.221+0000 7f0dfeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:31.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.228+0000 7f0dfeffd640 10 client.?.objecter ms_handle_connect 0x7f0e1014be90 2024-03-25T18:35:31.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.228+0000 7f0dfeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:31.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.228+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138207629 0 0) 0x7f0e101454b0 con 0x7f0e100b4d60 2024-03-25T18:35:31.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.228+0000 7f0dfeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:31.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.228+0000 7f0dfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:31.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.228+0000 7f0dfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:31.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.228+0000 7f0dfeffd640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:35:31.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.228+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0df0004eb0 con 0x7f0e100b4d60 2024-03-25T18:35:31.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.232+0000 7f0dfeffd640 10 client.?.objecter ms_handle_connect 0x7f0e100b1730 2024-03-25T18:35:31.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.232+0000 7f0dfeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:31.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.233+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728245094 0 0) 0x7f0e10143de0 con 0x7f0e1014be90 2024-03-25T18:35:31.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.233+0000 7f0dfeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:31.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.233+0000 7f0dfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:31.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.233+0000 7f0dfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:31.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.233+0000 7f0dfeffd640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:35:31.235 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.233+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e101454b0 con 0x7f0e1014be90 2024-03-25T18:35:31.237 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.236+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209342421 0 0) 0x7f0e101442a0 con 0x7f0e100b1730 2024-03-25T18:35:31.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.236+0000 7f0dfeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:31.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.236+0000 7f0dfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:31.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.236+0000 7f0dfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:31.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.236+0000 7f0dfeffd640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:35:31.238 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.236+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e10143de0 con 0x7f0e100b1730 2024-03-25T18:35:31.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.238+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4133412685 0 0) 0x7f0df0004eb0 con 0x7f0e100b4d60 2024-03-25T18:35:31.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.238+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e101442a0 con 0x7f0e100b4d60 2024-03-25T18:35:31.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.238+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0e04001b70 con 0x7f0e100b4d60 2024-03-25T18:35:31.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.238+0000 7f0dfeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:31.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.238+0000 7f0dfeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:31.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.238+0000 7f0dfeffd640 10 monclient(hunting): dump: 2024-03-25T18:35:31.239 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:31.239 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:31.239 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:31.239 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:31.239 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:31.239 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:31.239 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:31.240 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:31.240 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:31.240 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:31.240 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.238+0000 7f0dfeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:31.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.244+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3452941637 0 0) 0x7f0e10143de0 con 0x7f0e100b1730 2024-03-25T18:35:31.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.244+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0df0004eb0 con 0x7f0e100b1730 2024-03-25T18:35:31.246 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.244+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0e000017d0 con 0x7f0e100b1730 2024-03-25T18:35:31.246 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.244+0000 7f0dfeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:31.246 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.244+0000 7f0dfeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:31.246 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.244+0000 7f0dfeffd640 10 monclient(hunting): dump: 2024-03-25T18:35:31.246 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:31.246 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:31.246 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:31.246 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:31.246 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:31.246 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:31.246 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:31.246 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:31.246 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:31.246 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:31.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1825760884 0 0) 0x7f0e101442a0 con 0x7f0e100b4d60 2024-03-25T18:35:31.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 >> v1:172.21.15.99:6790/0 conn(0x7f0e100b1730 legacy=0x7f0e101d76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:31.248 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 >> v1:172.21.15.99:6789/0 conn(0x7f0e1014be90 legacy=0x7f0e101e2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0dfeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0dfeffd640 1 monclient: found mon.b 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0dfeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0dfeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e10144430 con 0x7f0e100b4d60 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0dfeffd640 10 monclient: _finish_auth 0 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0dfeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0dfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0e14df6640 5 monclient: authenticate success, global_id 4684 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0e14df6640 10 monclient: _renew_subs 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0e14df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0e14df6640 1 -- 172.21.15.99:0/2707245030 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1013e7a0 con 0x7f0e100b4d60 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0e14df6640 10 client.4684.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0e14df6640 10 monclient: _renew_subs 2024-03-25T18:35:31.249 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0e14df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:31.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0e14df6640 1 -- 172.21.15.99:0/2707245030 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e101d7fc0 con 0x7f0e100b4d60 2024-03-25T18:35:31.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.247+0000 7f0e14df6640 1 librados: init done 2024-03-25T18:35:31.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.248+0000 7f0e14df6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:31.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.248+0000 7f0e14df6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:31.250 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.248+0000 7f0e14df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:31.251 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.248+0000 7f0e14df6640 1 -- 172.21.15.99:0/2707245030 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e101452d0 con 0x7f0e100b4d60 2024-03-25T18:35:31.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.285+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e04004bf0 con 0x7f0e100b4d60 2024-03-25T18:35:31.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.285+0000 7f0dfeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:31.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.285+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f0e04004f00 con 0x7f0e100b4d60 2024-03-25T18:35:31.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.285+0000 7f0dfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:31.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.285+0000 7f0dfeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:31.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.285+0000 7f0dfeffd640 10 monclient: dump: 2024-03-25T18:35:31.287 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:31.287 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:31.287 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:31.287 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:31.287 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:31.288 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:31.288 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:31.288 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:31.288 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:31.288 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:31.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.285+0000 7f0e0cd99640 4 set_mon_vals no callback set 2024-03-25T18:35:31.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.286+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f0e0401aed0 con 0x7f0e100b4d60 2024-03-25T18:35:31.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.287+0000 7f0e0e59c640 10 monclient: get_auth_request con 0x7f0df006df30 auth_method 0 2024-03-25T18:35:31.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.287+0000 7f0dfeffd640 10 client.4684.objecter ms_handle_connect 0x7f0df006df30 2024-03-25T18:35:31.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.289+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2833420883 0 0) 0x7f0e040549f0 con 0x7f0e100b4d60 2024-03-25T18:35:31.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.289+0000 7f0dfeffd640 10 client.4684.objecter ms_dispatch 0x7f0e10000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:31.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.289+0000 7f0dfeffd640 3 client.4684.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:31.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.289+0000 7f0dfeffd640 3 client.4684.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:31.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.289+0000 7f0dfeffd640 20 client.4684.objecter dump_active .. 0 homeless 2024-03-25T18:35:31.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.560+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e04050500 con 0x7f0e100b4d60 2024-03-25T18:35:31.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.560+0000 7f0dfeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:31.561 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.560+0000 7f0dfeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:31.807 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.806+0000 7f0e14df6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:31.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.806+0000 7f0e14df6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:31.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.806+0000 7f0e14df6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:31.808 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.806+0000 7f0e14df6640 1 -- 172.21.15.99:0/2707245030 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e100627b0 con 0x7f0e100b4d60 2024-03-25T18:35:31.838 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.837+0000 7f0dfeffd640 1 -- 172.21.15.99:0/2707245030 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 270404165) 0x7f0e04052430 con 0x7f0e100b4d60 2024-03-25T18:35:31.838 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.837+0000 7f0dfeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:31.839 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.837+0000 7f0dfeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:31.839 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:35:31.839 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":11,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:35:20.161948+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:20.161948+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:31.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.840+0000 7f0e14df6640 10 librados: watch_flush enter 2024-03-25T18:35:31.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.840+0000 7f0e14df6640 10 librados: watch_flush exit 2024-03-25T18:35:31.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.840+0000 7f0e14df6640 20 client.4684.objecter shutdown clearing up homeless session... 2024-03-25T18:35:31.842 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.840+0000 7f0e14df6640 10 client.4684.objecter successfully canceled tick 2024-03-25T18:35:31.842 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.840+0000 7f0e14df6640 1 -- 172.21.15.99:0/2707245030 >> v1:172.21.15.190:6800/4220409699 conn(0x7f0df006df30 legacy=0x7f0df00703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:31.842 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.840+0000 7f0e14df6640 10 monclient: shutdown 2024-03-25T18:35:31.842 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.840+0000 7f0e14df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:31.842 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.840+0000 7f0e14df6640 1 -- 172.21.15.99:0/2707245030 >> v1:172.21.15.190:6789/0 conn(0x7f0e100b4d60 legacy=0x7f0e1012fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:31.842 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.840+0000 7f0e14df6640 1 -- 172.21.15.99:0/2707245030 shutdown_connections 2024-03-25T18:35:31.842 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.840+0000 7f0e14df6640 1 -- 172.21.15.99:0/2707245030 >> 172.21.15.99:0/2707245030 conn(0x7f0e100617a0 msgr2=0x7f0e100b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:31.842 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.840+0000 7f0e14df6640 1 -- 172.21.15.99:0/2707245030 shutdown_connections 2024-03-25T18:35:31.842 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.840+0000 7f0e14df6640 1 -- 172.21.15.99:0/2707245030 wait complete. 2024-03-25T18:35:31.842 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:31.840+0000 7f0e14df6640 1 librados: shutdown 2024-03-25T18:35:31.854 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:35.326 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:10:35.918 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:35.331 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:10:35.924 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:35.363 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:10:35.956 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:35.380 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:10:35.939 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:35.385 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:10:35.978 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:35.390 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:10:35.949 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:35.396 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:10:35.989 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:35.413 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:10:35.972 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:35.428 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:10:36.021 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:38.660 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:38.777 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.776+0000 7f56aee3e640 10 monclient: get_monmap_and_config 2024-03-25T18:35:38.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.776+0000 7f56aee3e640 10 monclient: build_initial_monmap 2024-03-25T18:35:38.778 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.776+0000 7f56aee3e640 10 monclient: monmap: 2024-03-25T18:35:38.778 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:38.778 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:38.778 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:35:38.777778+0000 2024-03-25T18:35:38.778 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:35:38.777778+0000 2024-03-25T18:35:38.778 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:38.778 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:38.778 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:38.778 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:35:38.778 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:35:38.778 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:38.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.777+0000 7f56aee3e640 1 Processor -- start 2024-03-25T18:35:38.779 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.777+0000 7f56aee3e640 1 -- start start 2024-03-25T18:35:38.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.777+0000 7f56aee3e640 10 monclient: init 2024-03-25T18:35:38.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.778+0000 7f56aee3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:38.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.778+0000 7f56aee3e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:38.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.778+0000 7f56aee3e640 10 monclient(hunting): picked mon.noname-c con 0x7f56a80a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:38.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.778+0000 7f56aee3e640 10 monclient(hunting): picked mon.noname-b con 0x7f56a813f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:38.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.778+0000 7f56aee3e640 10 monclient(hunting): picked mon.noname-a con 0x7f56a81430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:38.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.778+0000 7f56aee3e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a8142520 con 0x7f56a81430c0 2024-03-25T18:35:38.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.778+0000 7f56aee3e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a80b28a0 con 0x7f56a80a98f0 2024-03-25T18:35:38.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.778+0000 7f56aee3e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a80b2a30 con 0x7f56a813f710 2024-03-25T18:35:38.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.778+0000 7f56aee3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:38.780 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.778+0000 7f56aee3e640 10 monclient(hunting): authenticate will time out at 7956.946642s 2024-03-25T18:35:38.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.784+0000 7f56a7fff640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f56a813f710 0x7f56a81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:33618/0 (socket says 172.21.15.99:33618) 2024-03-25T18:35:38.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.784+0000 7f56a7fff640 1 -- 172.21.15.99:0/252594382 learned_addr learned my addr 172.21.15.99:0/252594382 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:38.799 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.798+0000 7f56a77fe640 1 -- 172.21.15.99:0/252594382 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749656117 0 0) 0x7f56a80b2a30 con 0x7f56a813f710 2024-03-25T18:35:38.799 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.798+0000 7f56a77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:38.800 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.798+0000 7f56a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:38.800 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.798+0000 7f56a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:38.800 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.798+0000 7f56a77fe640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:35:38.800 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.798+0000 7f56a77fe640 1 -- 172.21.15.99:0/252594382 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5698003520 con 0x7f56a813f710 2024-03-25T18:35:38.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.801+0000 7f56a77fe640 1 -- 172.21.15.99:0/252594382 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197905880 0 0) 0x7f56a80b28a0 con 0x7f56a80a98f0 2024-03-25T18:35:38.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.801+0000 7f56a77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:38.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.801+0000 7f56a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:38.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.801+0000 7f56a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:38.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.801+0000 7f56a77fe640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:35:38.802 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.801+0000 7f56a77fe640 1 -- 172.21.15.99:0/252594382 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a80b2a30 con 0x7f56a80a98f0 2024-03-25T18:35:38.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.808+0000 7f56a77fe640 1 -- 172.21.15.99:0/252594382 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241122579 0 0) 0x7f56a8142520 con 0x7f56a81430c0 2024-03-25T18:35:38.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.808+0000 7f56a77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:38.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.808+0000 7f56a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:38.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.808+0000 7f56a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:38.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.808+0000 7f56a77fe640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:35:38.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.808+0000 7f56a77fe640 1 -- 172.21.15.99:0/252594382 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a80b28a0 con 0x7f56a81430c0 2024-03-25T18:35:38.810 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 1 -- 172.21.15.99:0/252594382 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2633924365 0 0) 0x7f5698003520 con 0x7f56a813f710 2024-03-25T18:35:38.810 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 1 -- 172.21.15.99:0/252594382 >> v1:172.21.15.190:6789/0 conn(0x7f56a80a98f0 legacy=0x7f56a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:38.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 1 -- 172.21.15.99:0/252594382 >> v1:172.21.15.99:6789/0 conn(0x7f56a81430c0 legacy=0x7f56a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:38.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:35:38.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 1 monclient: found mon.noname-b 2024-03-25T18:35:38.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:38.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:35:38.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 1 -- 172.21.15.99:0/252594382 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a80b44c0 con 0x7f56a813f710 2024-03-25T18:35:38.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 10 monclient: _finish_auth 0 2024-03-25T18:35:38.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:35:38.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:38.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 1 -- 172.21.15.99:0/252594382 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5694001690 con 0x7f56a813f710 2024-03-25T18:35:38.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56aee3e640 5 monclient: authenticate success, global_id 4685 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56aee3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56a77fe640 10 monclient: dump: 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:38.812 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.809+0000 7f56aee3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:38.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.819+0000 7f56a77fe640 1 -- 172.21.15.99:0/252594382 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56940017f0 con 0x7f56a813f710 2024-03-25T18:35:38.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.819+0000 7f56a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:38.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.819+0000 7f56a77fe640 1 -- 172.21.15.99:0/252594382 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5694001c00 con 0x7f56a813f710 2024-03-25T18:35:38.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.819+0000 7f56aee3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:38.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.819+0000 7f56aee3e640 4 set_mon_vals no callback set 2024-03-25T18:35:38.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.819+0000 7f56a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:38.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.819+0000 7f56a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:38.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.819+0000 7f56a77fe640 10 monclient: dump: 2024-03-25T18:35:38.820 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:38.820 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:38.820 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:38.820 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:38.821 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:38.821 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:38.821 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:38.821 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:38.821 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:38.821 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:38.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.819+0000 7f56aee3e640 10 monclient: shutdown 2024-03-25T18:35:38.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.819+0000 7f56aee3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:38.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.819+0000 7f56aee3e640 1 -- 172.21.15.99:0/252594382 >> v1:172.21.15.99:6790/0 conn(0x7f56a813f710 legacy=0x7f56a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:38.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.819+0000 7f56aee3e640 1 -- 172.21.15.99:0/252594382 shutdown_connections 2024-03-25T18:35:38.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.820+0000 7f56aee3e640 1 -- 172.21.15.99:0/252594382 >> 172.21.15.99:0/252594382 conn(0x7f56a80b7ea0 msgr2=0x7f56a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:38.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.820+0000 7f56aee3e640 1 -- 172.21.15.99:0/252594382 shutdown_connections 2024-03-25T18:35:38.823 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.820+0000 7f56aee3e640 1 -- 172.21.15.99:0/252594382 wait complete. 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.820+0000 7f56aee3e640 10 monclient: build_initial_monmap 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.820+0000 7f56aee3e640 10 monclient: monmap: 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.821+0000 7f56aee3e640 1 librados: starting msgr at 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.821+0000 7f56aee3e640 1 librados: starting objecter 2024-03-25T18:35:38.824 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.821+0000 7f56aee3e640 1 Processor -- start 2024-03-25T18:35:38.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.821+0000 7f56aee3e640 1 -- start start 2024-03-25T18:35:38.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.821+0000 7f56aee3e640 1 librados: setting wanted keys 2024-03-25T18:35:38.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.821+0000 7f56aee3e640 1 librados: calling monclient init 2024-03-25T18:35:38.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.821+0000 7f56aee3e640 10 monclient: init 2024-03-25T18:35:38.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.822+0000 7f56aee3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:38.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.822+0000 7f56aee3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:38.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.822+0000 7f56aee3e640 10 monclient(hunting): picked mon.noname-c con 0x7f56a80a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:38.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.822+0000 7f56aee3e640 10 monclient(hunting): picked mon.noname-b con 0x7f56a813f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:38.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.822+0000 7f56aee3e640 10 monclient(hunting): picked mon.noname-a con 0x7f56a81430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:38.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.822+0000 7f56aee3e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a80abe70 con 0x7f56a81430c0 2024-03-25T18:35:38.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.822+0000 7f56aee3e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a8138280 con 0x7f56a813f710 2024-03-25T18:35:38.825 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.822+0000 7f56aee3e640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a81d51f0 con 0x7f56a80a98f0 2024-03-25T18:35:38.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.822+0000 7f56aee3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:38.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.822+0000 7f56aee3e640 10 monclient(hunting): authenticate will time out at 7956.990067s 2024-03-25T18:35:38.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.825+0000 7f56acbda640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f56a80a98f0 0x7f56a81387a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:33664/0 (socket says 172.21.15.99:33664) 2024-03-25T18:35:38.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.825+0000 7f56acbda640 1 -- 172.21.15.99:0/1230285040 learned_addr learned my addr 172.21.15.99:0/1230285040 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:38.827 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.825+0000 7f56a57fa640 10 client.?.objecter ms_handle_connect 0x7f56a80a98f0 2024-03-25T18:35:38.827 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.825+0000 7f56a57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:38.836 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.835+0000 7f56a57fa640 10 client.?.objecter ms_handle_connect 0x7f56a813f710 2024-03-25T18:35:38.836 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.835+0000 7f56a57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:38.842 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.841+0000 7f56a57fa640 10 client.?.objecter ms_handle_connect 0x7f56a81430c0 2024-03-25T18:35:38.842 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.841+0000 7f56a57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:38.858 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.857+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959452244 0 0) 0x7f56a80abe70 con 0x7f56a81430c0 2024-03-25T18:35:38.858 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.857+0000 7f56a57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:38.858 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.857+0000 7f56a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:38.858 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.857+0000 7f56a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:38.858 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.857+0000 7f56a57fa640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:35:38.858 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.857+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f568c004eb0 con 0x7f56a81430c0 2024-03-25T18:35:38.930 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.929+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1808315659 0 0) 0x7f568c004eb0 con 0x7f56a81430c0 2024-03-25T18:35:38.930 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.929+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56a80abe70 con 0x7f56a81430c0 2024-03-25T18:35:38.930 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.929+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f56a0001bd0 con 0x7f56a81430c0 2024-03-25T18:35:38.930 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.929+0000 7f56a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:38.930 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.929+0000 7f56a57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:38.930 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.929+0000 7f56a57fa640 10 monclient(hunting): dump: 2024-03-25T18:35:38.930 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:38.931 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:38.931 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:38.931 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:38.931 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:38.931 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:38.931 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:38.931 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:38.931 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:38.931 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:38.931 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.929+0000 7f56a57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:38.935 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.934+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511860763 0 0) 0x7f56a8138280 con 0x7f56a813f710 2024-03-25T18:35:38.935 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.934+0000 7f56a57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:38.935 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.934+0000 7f56a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:38.935 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.934+0000 7f56a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:38.936 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.934+0000 7f56a57fa640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:35:38.936 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.934+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f568c004eb0 con 0x7f56a813f710 2024-03-25T18:35:38.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.943+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313593178 0 0) 0x7f56a81d51f0 con 0x7f56a80a98f0 2024-03-25T18:35:38.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.943+0000 7f56a57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:38.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.943+0000 7f56a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:38.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.943+0000 7f56a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:38.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.943+0000 7f56a57fa640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:35:38.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.943+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a8138280 con 0x7f56a80a98f0 2024-03-25T18:35:38.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.949+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1279958751 0 0) 0x7f568c004eb0 con 0x7f56a813f710 2024-03-25T18:35:38.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.949+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56a81d51f0 con 0x7f56a813f710 2024-03-25T18:35:38.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.949+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f5694001b00 con 0x7f56a813f710 2024-03-25T18:35:38.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.949+0000 7f56a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:38.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.949+0000 7f56a57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:38.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.949+0000 7f56a57fa640 10 monclient(hunting): dump: 2024-03-25T18:35:38.951 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:38.951 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:38.951 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:38.951 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:38.951 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:38.951 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:38.951 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:38.951 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:38.951 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:38.951 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:38.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 77874078 0 0) 0x7f56a80abe70 con 0x7f56a81430c0 2024-03-25T18:35:38.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 >> v1:172.21.15.99:6790/0 conn(0x7f56a80a98f0 legacy=0x7f56a81387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:38.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 >> v1:172.21.15.190:6789/0 conn(0x7f56a813f710 legacy=0x7f56a81dade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:38.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56a57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:35:38.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56a57fa640 1 monclient: found mon.a 2024-03-25T18:35:38.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56a57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:38.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56a57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:35:38.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a8138eb0 con 0x7f56a81430c0 2024-03-25T18:35:38.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56a57fa640 10 monclient: _finish_auth 0 2024-03-25T18:35:38.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56a57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:35:38.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:38.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56aee3e640 5 monclient: authenticate success, global_id 4680 2024-03-25T18:35:38.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56aee3e640 10 monclient: _renew_subs 2024-03-25T18:35:38.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56aee3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:35:38.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56aee3e640 1 -- 172.21.15.99:0/1230285040 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a8139160 con 0x7f56a81430c0 2024-03-25T18:35:38.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56aee3e640 10 client.4680.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:38.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56aee3e640 10 monclient: _renew_subs 2024-03-25T18:35:38.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56aee3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:35:38.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56aee3e640 1 -- 172.21.15.99:0/1230285040 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a81e30e0 con 0x7f56a81430c0 2024-03-25T18:35:38.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.963+0000 7f56aee3e640 1 librados: init done 2024-03-25T18:35:38.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.964+0000 7f56aee3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:38.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.964+0000 7f56aee3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:38.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.964+0000 7f56aee3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:35:38.967 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:38.964+0000 7f56aee3e640 1 -- 172.21.15.99:0/1230285040 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a80627b0 con 0x7f56a81430c0 2024-03-25T18:35:39.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.281+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56a00020a0 con 0x7f56a81430c0 2024-03-25T18:35:39.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.281+0000 7f56a57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:39.282 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.281+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f56a0003b30 con 0x7f56a81430c0 2024-03-25T18:35:39.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.281+0000 7f56a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:39.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.281+0000 7f56a57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:39.283 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.281+0000 7f56a57fa640 10 monclient: dump: 2024-03-25T18:35:39.283 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:39.283 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:39.283 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:39.283 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:39.283 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:39.283 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:39.283 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:39.283 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:39.283 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:39.284 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:39.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.281+0000 7f56a6ffd640 4 set_mon_vals no callback set 2024-03-25T18:35:39.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.282+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f56a0003d50 con 0x7f56a81430c0 2024-03-25T18:35:39.284 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.282+0000 7f56acbda640 10 monclient: get_auth_request con 0x7f568c06dff0 auth_method 0 2024-03-25T18:35:39.285 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.283+0000 7f56a57fa640 10 client.4680.objecter ms_handle_connect 0x7f568c06dff0 2024-03-25T18:35:39.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.289+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2833420883 0 0) 0x7f56a00538a0 con 0x7f56a81430c0 2024-03-25T18:35:39.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.289+0000 7f56a57fa640 10 client.4680.objecter ms_dispatch 0x7f56a8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:39.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.289+0000 7f56a57fa640 3 client.4680.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:39.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.289+0000 7f56a57fa640 3 client.4680.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:39.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.289+0000 7f56a57fa640 20 client.4680.objecter dump_active .. 0 homeless 2024-03-25T18:35:39.290 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.289+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56a004f3b0 con 0x7f56a81430c0 2024-03-25T18:35:39.291 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.289+0000 7f56a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:39.291 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.289+0000 7f56a57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:39.543 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.541+0000 7f56aee3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:39.543 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.541+0000 7f56aee3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:39.543 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.541+0000 7f56aee3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:35:39.543 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.541+0000 7f56aee3e640 1 -- 172.21.15.99:0/1230285040 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56a8138410 con 0x7f56a81430c0 2024-03-25T18:35:39.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.595+0000 7f56a57fa640 1 -- 172.21.15.99:0/1230285040 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 270404165) 0x7f56a00512e0 con 0x7f56a81430c0 2024-03-25T18:35:39.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.595+0000 7f56a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:39.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.595+0000 7f56a57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:39.597 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:35:39.597 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":11,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:35:20.161948+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:20.161948+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:39.599 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.598+0000 7f56aee3e640 10 librados: watch_flush enter 2024-03-25T18:35:39.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.598+0000 7f56aee3e640 10 librados: watch_flush exit 2024-03-25T18:35:39.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.598+0000 7f56aee3e640 20 client.4680.objecter shutdown clearing up homeless session... 2024-03-25T18:35:39.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.598+0000 7f56aee3e640 10 client.4680.objecter successfully canceled tick 2024-03-25T18:35:39.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.598+0000 7f56aee3e640 1 -- 172.21.15.99:0/1230285040 >> v1:172.21.15.190:6800/4220409699 conn(0x7f568c06dff0 legacy=0x7f568c0704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:39.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.598+0000 7f56aee3e640 10 monclient: shutdown 2024-03-25T18:35:39.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.598+0000 7f56aee3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:39.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.598+0000 7f56aee3e640 1 -- 172.21.15.99:0/1230285040 >> v1:172.21.15.99:6789/0 conn(0x7f56a81430c0 legacy=0x7f56a81db4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:39.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.598+0000 7f56aee3e640 1 -- 172.21.15.99:0/1230285040 shutdown_connections 2024-03-25T18:35:39.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.598+0000 7f56aee3e640 1 -- 172.21.15.99:0/1230285040 >> 172.21.15.99:0/1230285040 conn(0x7f56a80b7ea0 msgr2=0x7f56a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:39.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.598+0000 7f56aee3e640 1 -- 172.21.15.99:0/1230285040 shutdown_connections 2024-03-25T18:35:39.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.598+0000 7f56aee3e640 1 -- 172.21.15.99:0/1230285040 wait complete. 2024-03-25T18:35:39.600 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:39.598+0000 7f56aee3e640 1 librados: shutdown 2024-03-25T18:35:39.609 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:46.414 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:46.528 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.527+0000 7f6085686640 10 monclient: get_monmap_and_config 2024-03-25T18:35:46.528 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.527+0000 7f6085686640 10 monclient: build_initial_monmap 2024-03-25T18:35:46.528 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.527+0000 7f6085686640 10 monclient: monmap: 2024-03-25T18:35:46.528 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:46.528 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:46.528 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:35:46.528173+0000 2024-03-25T18:35:46.528 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:35:46.528173+0000 2024-03-25T18:35:46.528 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:46.528 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:46.528 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:46.528 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:35:46.528 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:35:46.529 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:46.530 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.527+0000 7f6085686640 1 Processor -- start 2024-03-25T18:35:46.530 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.527+0000 7f6085686640 1 -- start start 2024-03-25T18:35:46.530 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.527+0000 7f6085686640 10 monclient: init 2024-03-25T18:35:46.530 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.528+0000 7f6085686640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:46.530 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.528+0000 7f6085686640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:46.530 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.528+0000 7f6085686640 10 monclient(hunting): picked mon.noname-a con 0x7f60800b4d60 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:46.530 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.528+0000 7f6085686640 10 monclient(hunting): picked mon.noname-b con 0x7f60800b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:46.530 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.528+0000 7f6085686640 10 monclient(hunting): picked mon.noname-c con 0x7f608014be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:46.530 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.528+0000 7f6085686640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60800af940 con 0x7f60800b4d60 2024-03-25T18:35:46.530 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.528+0000 7f6085686640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60800b0a70 con 0x7f608014be90 2024-03-25T18:35:46.530 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.528+0000 7f6085686640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60800b0d50 con 0x7f60800b1730 2024-03-25T18:35:46.530 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.528+0000 7f6085686640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:46.531 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.528+0000 7f6085686640 10 monclient(hunting): authenticate will time out at 7964.696415s 2024-03-25T18:35:46.537 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.536+0000 7f607e7fc640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f60800b1730 0x7f60800b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:44750/0 (socket says 172.21.15.99:44750) 2024-03-25T18:35:46.537 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.536+0000 7f607e7fc640 1 -- 172.21.15.99:0/3040612971 learned_addr learned my addr 172.21.15.99:0/3040612971 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:46.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.548+0000 7f607dffb640 1 -- 172.21.15.99:0/3040612971 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532097379 0 0) 0x7f60800b0d50 con 0x7f60800b1730 2024-03-25T18:35:46.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.548+0000 7f607dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:46.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.548+0000 7f607dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:46.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.548+0000 7f607dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:46.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.548+0000 7f607dffb640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:35:46.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.548+0000 7f607dffb640 1 -- 172.21.15.99:0/3040612971 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6068003520 con 0x7f60800b1730 2024-03-25T18:35:46.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.555+0000 7f607dffb640 1 -- 172.21.15.99:0/3040612971 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845184410 0 0) 0x7f60800b0a70 con 0x7f608014be90 2024-03-25T18:35:46.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.555+0000 7f607dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:46.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.555+0000 7f607dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:46.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.555+0000 7f607dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:46.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.555+0000 7f607dffb640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:35:46.556 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.555+0000 7f607dffb640 1 -- 172.21.15.99:0/3040612971 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60800b0d50 con 0x7f608014be90 2024-03-25T18:35:46.568 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.567+0000 7f607dffb640 1 -- 172.21.15.99:0/3040612971 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755889771 0 0) 0x7f60800af940 con 0x7f60800b4d60 2024-03-25T18:35:46.568 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.567+0000 7f607dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:46.568 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.567+0000 7f607dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:46.568 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.567+0000 7f607dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:46.568 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.567+0000 7f607dffb640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:35:46.568 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.567+0000 7f607dffb640 1 -- 172.21.15.99:0/3040612971 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60800b0a70 con 0x7f60800b4d60 2024-03-25T18:35:46.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 1 -- 172.21.15.99:0/3040612971 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3087071620 0 0) 0x7f6068003520 con 0x7f60800b1730 2024-03-25T18:35:46.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 1 -- 172.21.15.99:0/3040612971 >> v1:172.21.15.190:6789/0 conn(0x7f608014be90 legacy=0x7f608014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:46.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 1 -- 172.21.15.99:0/3040612971 >> v1:172.21.15.99:6789/0 conn(0x7f60800b4d60 legacy=0x7f608013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:46.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:35:46.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 1 monclient: found mon.noname-b 2024-03-25T18:35:46.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:46.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:35:46.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 1 -- 172.21.15.99:0/3040612971 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60800b1030 con 0x7f60800b1730 2024-03-25T18:35:46.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 10 monclient: _finish_auth 0 2024-03-25T18:35:46.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 1 -- 172.21.15.99:0/3040612971 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6074001690 con 0x7f60800b1730 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f607dffb640 10 monclient: dump: 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:46.579 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:46.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f6085686640 5 monclient: authenticate success, global_id 4691 2024-03-25T18:35:46.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.577+0000 7f6085686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:46.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.589+0000 7f607dffb640 1 -- 172.21.15.99:0/3040612971 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60740017f0 con 0x7f60800b1730 2024-03-25T18:35:46.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.589+0000 7f607dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:46.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.589+0000 7f607dffb640 1 -- 172.21.15.99:0/3040612971 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6074001c00 con 0x7f60800b1730 2024-03-25T18:35:46.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.589+0000 7f6085686640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:46.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.589+0000 7f6085686640 4 set_mon_vals no callback set 2024-03-25T18:35:46.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.589+0000 7f607dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:46.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.589+0000 7f607dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:46.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.589+0000 7f607dffb640 10 monclient: dump: 2024-03-25T18:35:46.590 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:46.590 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:46.590 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:46.591 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:46.591 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:46.591 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:46.591 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:46.591 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:46.591 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:46.591 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:46.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.590+0000 7f6085686640 10 monclient: shutdown 2024-03-25T18:35:46.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.590+0000 7f6085686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:46.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.590+0000 7f6085686640 1 -- 172.21.15.99:0/3040612971 >> v1:172.21.15.99:6790/0 conn(0x7f60800b1730 legacy=0x7f60800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:46.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.590+0000 7f6085686640 1 -- 172.21.15.99:0/3040612971 shutdown_connections 2024-03-25T18:35:46.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.590+0000 7f6085686640 1 -- 172.21.15.99:0/3040612971 >> 172.21.15.99:0/3040612971 conn(0x7f60800617a0 msgr2=0x7f6080061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:46.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.590+0000 7f6085686640 1 -- 172.21.15.99:0/3040612971 shutdown_connections 2024-03-25T18:35:46.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.590+0000 7f6085686640 1 -- 172.21.15.99:0/3040612971 wait complete. 2024-03-25T18:35:46.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.590+0000 7f6085686640 10 monclient: build_initial_monmap 2024-03-25T18:35:46.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.590+0000 7f6085686640 10 monclient: monmap: 2024-03-25T18:35:46.594 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:46.594 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:46.594 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:35:46.594 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:35:46.594 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:46.594 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:35:46.594 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:46.594 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:35:46.594 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:35:46.594 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:46.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.591+0000 7f6085686640 1 librados: starting msgr at 2024-03-25T18:35:46.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.591+0000 7f6085686640 1 librados: starting objecter 2024-03-25T18:35:46.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.591+0000 7f6085686640 1 Processor -- start 2024-03-25T18:35:46.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.591+0000 7f6085686640 1 -- start start 2024-03-25T18:35:46.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.591+0000 7f6085686640 1 librados: setting wanted keys 2024-03-25T18:35:46.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.591+0000 7f6085686640 1 librados: calling monclient init 2024-03-25T18:35:46.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.591+0000 7f6085686640 10 monclient: init 2024-03-25T18:35:46.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.592+0000 7f6085686640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:46.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.592+0000 7f6085686640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:46.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.592+0000 7f6085686640 10 monclient(hunting): picked mon.noname-a con 0x7f60800b1730 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:46.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.592+0000 7f6085686640 10 monclient(hunting): picked mon.noname-b con 0x7f60800b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:46.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.592+0000 7f6085686640 10 monclient(hunting): picked mon.noname-c con 0x7f608014be90 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:46.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.592+0000 7f6085686640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60801d7840 con 0x7f60800b1730 2024-03-25T18:35:46.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.592+0000 7f6085686640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6080138860 con 0x7f60800b4d60 2024-03-25T18:35:46.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.592+0000 7f6085686640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60801389f0 con 0x7f608014be90 2024-03-25T18:35:46.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.592+0000 7f6085686640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:46.596 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.592+0000 7f6085686640 10 monclient(hunting): authenticate will time out at 7964.760067s 2024-03-25T18:35:46.602 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.601+0000 7f607f7fe640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f608014be90 0x7f6080142820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:44808/0 (socket says 172.21.15.99:44808) 2024-03-25T18:35:46.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.601+0000 7f607f7fe640 1 -- 172.21.15.99:0/2912065692 learned_addr learned my addr 172.21.15.99:0/2912065692 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:46.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.602+0000 7f60677fe640 10 client.?.objecter ms_handle_connect 0x7f608014be90 2024-03-25T18:35:46.603 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.602+0000 7f60677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:46.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.608+0000 7f60677fe640 10 client.?.objecter ms_handle_connect 0x7f60800b1730 2024-03-25T18:35:46.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.608+0000 7f60677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:46.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.608+0000 7f60677fe640 10 client.?.objecter ms_handle_connect 0x7f60800b4d60 2024-03-25T18:35:46.609 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.608+0000 7f60677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:46.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.609+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465743347 0 0) 0x7f60801389f0 con 0x7f608014be90 2024-03-25T18:35:46.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.609+0000 7f60677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:46.610 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.609+0000 7f60677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:46.611 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.609+0000 7f60677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:46.611 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.609+0000 7f60677fe640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:35:46.611 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.609+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f606c004eb0 con 0x7f608014be90 2024-03-25T18:35:46.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.620+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595294524 0 0) 0x7f6080138860 con 0x7f60800b4d60 2024-03-25T18:35:46.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.620+0000 7f60677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:46.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.620+0000 7f60677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:46.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.620+0000 7f60677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:46.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.620+0000 7f60677fe640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:35:46.621 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.620+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60801389f0 con 0x7f60800b4d60 2024-03-25T18:35:46.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.624+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807597555 0 0) 0x7f60801d7840 con 0x7f60800b1730 2024-03-25T18:35:46.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.624+0000 7f60677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:46.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.624+0000 7f60677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:46.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.624+0000 7f60677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:46.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.624+0000 7f60677fe640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:35:46.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.624+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6080138860 con 0x7f60800b1730 2024-03-25T18:35:46.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.625+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1519636597 0 0) 0x7f606c004eb0 con 0x7f608014be90 2024-03-25T18:35:46.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.625+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60801d7840 con 0x7f608014be90 2024-03-25T18:35:46.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.625+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6078001c90 con 0x7f608014be90 2024-03-25T18:35:46.626 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.625+0000 7f60677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:46.627 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.625+0000 7f60677fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:46.627 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.625+0000 7f60677fe640 10 monclient(hunting): dump: 2024-03-25T18:35:46.627 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:46.627 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:46.627 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:46.627 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:46.627 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:46.627 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:46.627 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:46.627 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:46.627 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:46.627 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:46.627 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.625+0000 7f60677fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:46.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 713058009 0 0) 0x7f60801d7840 con 0x7f608014be90 2024-03-25T18:35:46.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 >> v1:172.21.15.190:6789/0 conn(0x7f60800b4d60 legacy=0x7f60801d7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:46.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 >> v1:172.21.15.99:6789/0 conn(0x7f60800b1730 legacy=0x7f60801405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:46.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f60677fe640 10 monclient: _finish_hunting 0 2024-03-25T18:35:46.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f60677fe640 1 monclient: found mon.c 2024-03-25T18:35:46.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f60677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:46.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f60677fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:35:46.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60801d8430 con 0x7f608014be90 2024-03-25T18:35:46.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f60677fe640 10 monclient: _finish_auth 0 2024-03-25T18:35:46.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f60677fe640 10 monclient: _check_auth_tickets 2024-03-25T18:35:46.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f60677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:46.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f6085686640 5 monclient: authenticate success, global_id 4694 2024-03-25T18:35:46.637 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f6085686640 10 monclient: _renew_subs 2024-03-25T18:35:46.638 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f6085686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:35:46.638 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.635+0000 7f6085686640 1 -- 172.21.15.99:0/2912065692 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6080137e50 con 0x7f608014be90 2024-03-25T18:35:46.638 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.636+0000 7f6085686640 10 client.4694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:46.638 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.636+0000 7f6085686640 10 monclient: _renew_subs 2024-03-25T18:35:46.638 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.636+0000 7f6085686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:35:46.638 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.636+0000 7f6085686640 1 -- 172.21.15.99:0/2912065692 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6080142350 con 0x7f608014be90 2024-03-25T18:35:46.638 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.636+0000 7f6085686640 1 librados: init done 2024-03-25T18:35:46.638 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.637+0000 7f6085686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:46.639 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.637+0000 7f6085686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:46.639 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.637+0000 7f6085686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:35:46.639 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.637+0000 7f6085686640 1 -- 172.21.15.99:0/2912065692 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60801d86a0 con 0x7f608014be90 2024-03-25T18:35:46.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.941+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6078003970 con 0x7f608014be90 2024-03-25T18:35:46.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.941+0000 7f60677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:46.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.941+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6078003d80 con 0x7f608014be90 2024-03-25T18:35:46.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.941+0000 7f60677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:46.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.941+0000 7f60677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:46.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.941+0000 7f60677fe640 10 monclient: dump: 2024-03-25T18:35:46.943 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:46.943 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:46.943 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:46.943 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:46.943 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:46.943 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:46.943 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:46.944 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:46.944 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:46.944 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:46.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.941+0000 7f607d7fa640 4 set_mon_vals no callback set 2024-03-25T18:35:46.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.942+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f6078003fa0 con 0x7f608014be90 2024-03-25T18:35:46.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.943+0000 7f607effd640 10 monclient: get_auth_request con 0x7f606c06de20 auth_method 0 2024-03-25T18:35:46.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.943+0000 7f60677fe640 10 client.4694.objecter ms_handle_connect 0x7f606c06de20 2024-03-25T18:35:46.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.949+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2833420883 0 0) 0x7f6078054a20 con 0x7f608014be90 2024-03-25T18:35:46.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.949+0000 7f60677fe640 10 client.4694.objecter ms_dispatch 0x7f6080000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:46.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.949+0000 7f60677fe640 3 client.4694.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:46.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.949+0000 7f60677fe640 3 client.4694.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:46.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.949+0000 7f60677fe640 20 client.4694.objecter dump_active .. 0 homeless 2024-03-25T18:35:46.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.949+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6078050530 con 0x7f608014be90 2024-03-25T18:35:46.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.949+0000 7f60677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:46.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:46.949+0000 7f60677fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:47.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.175+0000 7f6085686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:47.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.175+0000 7f6085686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:47.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.175+0000 7f6085686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:35:47.176 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.175+0000 7f6085686640 1 -- 172.21.15.99:0/2912065692 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60800627b0 con 0x7f608014be90 2024-03-25T18:35:47.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.200+0000 7f60677fe640 1 -- 172.21.15.99:0/2912065692 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 270404165) 0x7f6078052460 con 0x7f608014be90 2024-03-25T18:35:47.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.200+0000 7f60677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:47.201 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.200+0000 7f60677fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:47.201 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:35:47.202 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":11,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:35:20.161948+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:20.161948+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:47.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.202+0000 7f6085686640 10 librados: watch_flush enter 2024-03-25T18:35:47.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.202+0000 7f6085686640 10 librados: watch_flush exit 2024-03-25T18:35:47.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.202+0000 7f6085686640 20 client.4694.objecter shutdown clearing up homeless session... 2024-03-25T18:35:47.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.202+0000 7f6085686640 10 client.4694.objecter successfully canceled tick 2024-03-25T18:35:47.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.202+0000 7f6085686640 1 -- 172.21.15.99:0/2912065692 >> v1:172.21.15.190:6800/4220409699 conn(0x7f606c06de20 legacy=0x7f606c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:47.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.202+0000 7f6085686640 10 monclient: shutdown 2024-03-25T18:35:47.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.202+0000 7f6085686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:47.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.202+0000 7f6085686640 1 -- 172.21.15.99:0/2912065692 >> v1:172.21.15.99:6790/0 conn(0x7f608014be90 legacy=0x7f6080142820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:47.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.202+0000 7f6085686640 1 -- 172.21.15.99:0/2912065692 shutdown_connections 2024-03-25T18:35:47.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.202+0000 7f6085686640 1 -- 172.21.15.99:0/2912065692 >> 172.21.15.99:0/2912065692 conn(0x7f60800617a0 msgr2=0x7f6080061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:47.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.202+0000 7f6085686640 1 -- 172.21.15.99:0/2912065692 shutdown_connections 2024-03-25T18:35:47.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.202+0000 7f6085686640 1 -- 172.21.15.99:0/2912065692 wait complete. 2024-03-25T18:35:47.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:47.202+0000 7f6085686640 1 librados: shutdown 2024-03-25T18:35:47.215 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:54.022 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:54.141 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.140+0000 7ff27448d640 10 monclient: get_monmap_and_config 2024-03-25T18:35:54.141 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.140+0000 7ff27448d640 10 monclient: build_initial_monmap 2024-03-25T18:35:54.141 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.140+0000 7ff27448d640 10 monclient: monmap: 2024-03-25T18:35:54.141 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:54.141 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:54.141 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:35:54.141133+0000 2024-03-25T18:35:54.141 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:35:54.141133+0000 2024-03-25T18:35:54.141 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:54.141 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:54.142 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:54.142 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:35:54.142 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:35:54.142 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:54.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.140+0000 7ff27448d640 1 Processor -- start 2024-03-25T18:35:54.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.140+0000 7ff27448d640 1 -- start start 2024-03-25T18:35:54.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.140+0000 7ff27448d640 10 monclient: init 2024-03-25T18:35:54.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.141+0000 7ff27448d640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:54.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.142+0000 7ff27448d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:54.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.142+0000 7ff27448d640 10 monclient(hunting): picked mon.noname-a con 0x7ff26c0a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:54.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.142+0000 7ff27448d640 10 monclient(hunting): picked mon.noname-b con 0x7ff26c13f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:54.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.142+0000 7ff27448d640 10 monclient(hunting): picked mon.noname-c con 0x7ff26c1430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:54.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.142+0000 7ff27448d640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c142520 con 0x7ff26c0a98f0 2024-03-25T18:35:54.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.142+0000 7ff27448d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c0b28a0 con 0x7ff26c1430c0 2024-03-25T18:35:54.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.142+0000 7ff27448d640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c0b2a30 con 0x7ff26c13f710 2024-03-25T18:35:54.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.142+0000 7ff27448d640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:54.144 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.142+0000 7ff27448d640 10 monclient(hunting): authenticate will time out at 7972.310719s 2024-03-25T18:35:54.148 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.147+0000 7ff272a2a640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ff26c1430c0 0x7ff26c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:41880/0 (socket says 172.21.15.99:41880) 2024-03-25T18:35:54.149 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.147+0000 7ff272a2a640 1 -- 172.21.15.99:0/3571629198 learned_addr learned my addr 172.21.15.99:0/3571629198 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:54.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.160+0000 7ff271227640 1 -- 172.21.15.99:0/3571629198 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940719433 0 0) 0x7ff26c0b28a0 con 0x7ff26c1430c0 2024-03-25T18:35:54.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.160+0000 7ff271227640 10 monclient(hunting): _negotiate 2024-03-25T18:35:54.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.160+0000 7ff271227640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:54.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.160+0000 7ff271227640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:54.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.160+0000 7ff271227640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:35:54.161 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.160+0000 7ff271227640 1 -- 172.21.15.99:0/3571629198 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff258003520 con 0x7ff26c1430c0 2024-03-25T18:35:54.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.171+0000 7ff271227640 1 -- 172.21.15.99:0/3571629198 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3649674552 0 0) 0x7ff26c0b2a30 con 0x7ff26c13f710 2024-03-25T18:35:54.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.171+0000 7ff271227640 10 monclient(hunting): _negotiate 2024-03-25T18:35:54.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.171+0000 7ff271227640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:54.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.171+0000 7ff271227640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:54.172 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.171+0000 7ff271227640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:35:54.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.171+0000 7ff271227640 1 -- 172.21.15.99:0/3571629198 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff26c0b28a0 con 0x7ff26c13f710 2024-03-25T18:35:54.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.190+0000 7ff271227640 1 -- 172.21.15.99:0/3571629198 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875083136 0 0) 0x7ff26c142520 con 0x7ff26c0a98f0 2024-03-25T18:35:54.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.190+0000 7ff271227640 10 monclient(hunting): _negotiate 2024-03-25T18:35:54.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.190+0000 7ff271227640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:54.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.190+0000 7ff271227640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:54.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.190+0000 7ff271227640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:35:54.192 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.190+0000 7ff271227640 1 -- 172.21.15.99:0/3571629198 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff26c0b2a30 con 0x7ff26c0a98f0 2024-03-25T18:35:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 1 -- 172.21.15.99:0/3571629198 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2958629763 0 0) 0x7ff258003520 con 0x7ff26c1430c0 2024-03-25T18:35:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 1 -- 172.21.15.99:0/3571629198 >> v1:172.21.15.99:6790/0 conn(0x7ff26c13f710 legacy=0x7ff26c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 1 -- 172.21.15.99:0/3571629198 >> v1:172.21.15.99:6789/0 conn(0x7ff26c0a98f0 legacy=0x7ff26c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 10 monclient: _finish_hunting 0 2024-03-25T18:35:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 1 monclient: found mon.noname-c 2024-03-25T18:35:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:35:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 1 -- 172.21.15.99:0/3571629198 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c0b44c0 con 0x7ff26c1430c0 2024-03-25T18:35:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 10 monclient: _finish_auth 0 2024-03-25T18:35:54.195 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 10 monclient: _check_auth_tickets 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 1 -- 172.21.15.99:0/3571629198 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff264001a10 con 0x7ff26c1430c0 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff271227640 10 monclient: dump: 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:54.196 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:54.197 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff27448d640 5 monclient: authenticate success, global_id 4699 2024-03-25T18:35:54.197 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.193+0000 7ff27448d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:54.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.221+0000 7ff271227640 1 -- 172.21.15.99:0/3571629198 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff264001b70 con 0x7ff26c1430c0 2024-03-25T18:35:54.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.221+0000 7ff271227640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:54.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.221+0000 7ff271227640 1 -- 172.21.15.99:0/3571629198 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff264001f80 con 0x7ff26c1430c0 2024-03-25T18:35:54.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.221+0000 7ff27448d640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:54.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.221+0000 7ff27448d640 4 set_mon_vals no callback set 2024-03-25T18:35:54.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.221+0000 7ff271227640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:54.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.221+0000 7ff271227640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:54.223 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.221+0000 7ff271227640 10 monclient: dump: 2024-03-25T18:35:54.224 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:54.224 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:54.224 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:54.224 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:54.224 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:54.224 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:54.224 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:54.224 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:54.224 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:54.224 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:54.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.222+0000 7ff27448d640 10 monclient: shutdown 2024-03-25T18:35:54.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.222+0000 7ff27448d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:54.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.222+0000 7ff27448d640 1 -- 172.21.15.99:0/3571629198 >> v1:172.21.15.190:6789/0 conn(0x7ff26c1430c0 legacy=0x7ff26c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:54.224 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.222+0000 7ff27448d640 1 -- 172.21.15.99:0/3571629198 shutdown_connections 2024-03-25T18:35:54.225 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.222+0000 7ff27448d640 1 -- 172.21.15.99:0/3571629198 >> 172.21.15.99:0/3571629198 conn(0x7ff26c0b7ea0 msgr2=0x7ff26c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:54.225 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.222+0000 7ff27448d640 1 -- 172.21.15.99:0/3571629198 shutdown_connections 2024-03-25T18:35:54.225 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.222+0000 7ff27448d640 1 -- 172.21.15.99:0/3571629198 wait complete. 2024-03-25T18:35:54.226 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.222+0000 7ff27448d640 10 monclient: build_initial_monmap 2024-03-25T18:35:54.226 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.222+0000 7ff27448d640 10 monclient: monmap: 2024-03-25T18:35:54.226 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:35:54.226 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:54.227 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:35:54.227 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:35:54.227 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:54.227 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:35:54.227 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:35:54.227 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:35:54.227 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:35:54.227 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:54.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.223+0000 7ff27448d640 1 librados: starting msgr at 2024-03-25T18:35:54.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.223+0000 7ff27448d640 1 librados: starting objecter 2024-03-25T18:35:54.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.223+0000 7ff27448d640 1 Processor -- start 2024-03-25T18:35:54.227 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.223+0000 7ff27448d640 1 -- start start 2024-03-25T18:35:54.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.223+0000 7ff27448d640 1 librados: setting wanted keys 2024-03-25T18:35:54.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.223+0000 7ff27448d640 1 librados: calling monclient init 2024-03-25T18:35:54.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.223+0000 7ff27448d640 10 monclient: init 2024-03-25T18:35:54.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.224+0000 7ff27448d640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:54.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.224+0000 7ff27448d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:54.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.224+0000 7ff27448d640 10 monclient(hunting): picked mon.noname-b con 0x7ff26c0a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:35:54.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.224+0000 7ff27448d640 10 monclient(hunting): picked mon.noname-a con 0x7ff26c13f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:35:54.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.224+0000 7ff27448d640 10 monclient(hunting): picked mon.noname-c con 0x7ff26c1430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:35:54.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.224+0000 7ff27448d640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c1d51f0 con 0x7ff26c13f710 2024-03-25T18:35:54.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.224+0000 7ff27448d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c1db8c0 con 0x7ff26c0a98f0 2024-03-25T18:35:54.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.224+0000 7ff27448d640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c1dac90 con 0x7ff26c1430c0 2024-03-25T18:35:54.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.224+0000 7ff27448d640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:54.228 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.224+0000 7ff27448d640 10 monclient(hunting): authenticate will time out at 7972.392136s 2024-03-25T18:35:54.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.226+0000 7ff272229640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ff26c0a98f0 0x7ff26c1db1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:41932/0 (socket says 172.21.15.99:41932) 2024-03-25T18:35:54.229 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.226+0000 7ff272229640 1 -- 172.21.15.99:0/1394065122 learned_addr learned my addr 172.21.15.99:0/1394065122 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:35:54.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.229+0000 7ff262ffd640 10 client.?.objecter ms_handle_connect 0x7ff26c0a98f0 2024-03-25T18:35:54.230 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.229+0000 7ff262ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:54.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.232+0000 7ff262ffd640 10 client.?.objecter ms_handle_connect 0x7ff26c1430c0 2024-03-25T18:35:54.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.232+0000 7ff262ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:54.241 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.240+0000 7ff262ffd640 10 client.?.objecter ms_handle_connect 0x7ff26c13f710 2024-03-25T18:35:54.241 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.240+0000 7ff262ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:54.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.242+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692068264 0 0) 0x7ff26c1db8c0 con 0x7ff26c0a98f0 2024-03-25T18:35:54.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.242+0000 7ff262ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:54.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.242+0000 7ff262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:54.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.242+0000 7ff262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:54.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.242+0000 7ff262ffd640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:35:54.243 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.242+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff250004eb0 con 0x7ff26c0a98f0 2024-03-25T18:35:54.246 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.245+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894915666 0 0) 0x7ff26c1dac90 con 0x7ff26c1430c0 2024-03-25T18:35:54.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.245+0000 7ff262ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:54.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.245+0000 7ff262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:54.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.245+0000 7ff262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:54.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.245+0000 7ff262ffd640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:35:54.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.246+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff26c1db8c0 con 0x7ff26c1430c0 2024-03-25T18:35:54.252 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.251+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123231232 0 0) 0x7ff26c1d51f0 con 0x7ff26c13f710 2024-03-25T18:35:54.252 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.251+0000 7ff262ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:54.252 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.251+0000 7ff262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:54.252 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.251+0000 7ff262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:54.252 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.251+0000 7ff262ffd640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:35:54.252 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.251+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff26c1dac90 con 0x7ff26c13f710 2024-03-25T18:35:54.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.268+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3378141009 0 0) 0x7ff26c1db8c0 con 0x7ff26c1430c0 2024-03-25T18:35:54.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.268+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff26c1d51f0 con 0x7ff26c1430c0 2024-03-25T18:35:54.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.268+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff264001e80 con 0x7ff26c1430c0 2024-03-25T18:35:54.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.268+0000 7ff262ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:54.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.268+0000 7ff262ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:54.270 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.268+0000 7ff262ffd640 10 monclient(hunting): dump: 2024-03-25T18:35:54.270 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:54.270 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:54.270 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:54.270 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.268+0000 7ff262ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.268+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1411558145 0 0) 0x7ff250004eb0 con 0x7ff26c0a98f0 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.268+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff26c1db8c0 con 0x7ff26c0a98f0 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.268+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff268001aa0 con 0x7ff26c0a98f0 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.268+0000 7ff262ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.268+0000 7ff262ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.268+0000 7ff262ffd640 10 monclient(hunting): dump: 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:54.271 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:54.272 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:54.272 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:54.272 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:54.272 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:54.272 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:54.272 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:54.272 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:54.272 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:54.276 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.275+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1537854382 0 0) 0x7ff26c1dac90 con 0x7ff26c13f710 2024-03-25T18:35:54.276 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.275+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff250004eb0 con 0x7ff26c13f710 2024-03-25T18:35:54.276 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.275+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff25c0018f0 con 0x7ff26c13f710 2024-03-25T18:35:54.277 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.275+0000 7ff262ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:54.277 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.275+0000 7ff262ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:54.277 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.275+0000 7ff262ffd640 10 monclient(hunting): dump: 2024-03-25T18:35:54.277 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:54.277 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:54.277 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:54.277 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:54.277 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:54.277 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:54.277 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:54.277 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:54.277 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:54.277 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:54.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 553848660 0 0) 0x7ff26c1db8c0 con 0x7ff26c0a98f0 2024-03-25T18:35:54.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 >> v1:172.21.15.99:6790/0 conn(0x7ff26c1430c0 legacy=0x7ff26c1e1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:54.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 >> v1:172.21.15.99:6789/0 conn(0x7ff26c13f710 legacy=0x7ff26c1385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:54.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff262ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:54.287 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff262ffd640 1 monclient: found mon.b 2024-03-25T18:35:54.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff262ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:54.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff262ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:54.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c1399c0 con 0x7ff26c0a98f0 2024-03-25T18:35:54.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff262ffd640 10 monclient: _finish_auth 0 2024-03-25T18:35:54.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff262ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:54.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:54.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff27448d640 5 monclient: authenticate success, global_id 4702 2024-03-25T18:35:54.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff27448d640 10 monclient: _renew_subs 2024-03-25T18:35:54.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff27448d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:54.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff27448d640 1 -- 172.21.15.99:0/1394065122 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff26c138cb0 con 0x7ff26c0a98f0 2024-03-25T18:35:54.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff27448d640 10 client.4702.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:54.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff27448d640 10 monclient: _renew_subs 2024-03-25T18:35:54.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff27448d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:54.288 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff27448d640 1 -- 172.21.15.99:0/1394065122 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff26c138050 con 0x7ff26c0a98f0 2024-03-25T18:35:54.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.286+0000 7ff27448d640 1 librados: init done 2024-03-25T18:35:54.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.287+0000 7ff27448d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:54.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.287+0000 7ff27448d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:54.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.287+0000 7ff27448d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:54.289 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.287+0000 7ff27448d640 1 -- 172.21.15.99:0/1394065122 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff26c0627b0 con 0x7ff26c0a98f0 2024-03-25T18:35:54.591 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.590+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff268001d00 con 0x7ff26c0a98f0 2024-03-25T18:35:54.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.590+0000 7ff262ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:54.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.590+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ff268003a60 con 0x7ff26c0a98f0 2024-03-25T18:35:54.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.590+0000 7ff262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:54.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.590+0000 7ff262ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:54.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.590+0000 7ff262ffd640 10 monclient: dump: 2024-03-25T18:35:54.592 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:35:54.592 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:35:54.592 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:54.592 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:35:54.592 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:54.592 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:35:54.592 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:35:54.592 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:35:54.592 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:35:54.593 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:35:54.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.590+0000 7ff270a26640 4 set_mon_vals no callback set 2024-03-25T18:35:54.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.591+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7ff268003c80 con 0x7ff26c0a98f0 2024-03-25T18:35:54.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.592+0000 7ff271a28640 10 monclient: get_auth_request con 0x7ff25006e0c0 auth_method 0 2024-03-25T18:35:54.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.592+0000 7ff262ffd640 10 client.4702.objecter ms_handle_connect 0x7ff25006e0c0 2024-03-25T18:35:54.597 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.596+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2833420883 0 0) 0x7ff2680548e0 con 0x7ff26c0a98f0 2024-03-25T18:35:54.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.596+0000 7ff262ffd640 10 client.4702.objecter ms_dispatch 0x7ff26c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:35:54.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.596+0000 7ff262ffd640 3 client.4702.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:35:54.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.596+0000 7ff262ffd640 3 client.4702.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:35:54.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.597+0000 7ff262ffd640 20 client.4702.objecter dump_active .. 0 homeless 2024-03-25T18:35:54.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.597+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff2680503f0 con 0x7ff26c0a98f0 2024-03-25T18:35:54.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.597+0000 7ff262ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:54.598 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.597+0000 7ff262ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:54.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.858+0000 7ff27448d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:54.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.858+0000 7ff27448d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:54.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.858+0000 7ff27448d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:35:54.860 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.858+0000 7ff27448d640 1 -- 172.21.15.99:0/1394065122 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff26c1dae20 con 0x7ff26c0a98f0 2024-03-25T18:35:54.892 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.891+0000 7ff262ffd640 1 -- 172.21.15.99:0/1394065122 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 270404165) 0x7ff268052320 con 0x7ff26c0a98f0 2024-03-25T18:35:54.892 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.891+0000 7ff262ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:54.892 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.891+0000 7ff262ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:54.892 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:35:54.892 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":11,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:35:20.161948+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:20.161948+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:54.895 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.893+0000 7ff27448d640 10 librados: watch_flush enter 2024-03-25T18:35:54.895 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.893+0000 7ff27448d640 10 librados: watch_flush exit 2024-03-25T18:35:54.895 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.893+0000 7ff27448d640 20 client.4702.objecter shutdown clearing up homeless session... 2024-03-25T18:35:54.895 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.893+0000 7ff27448d640 10 client.4702.objecter successfully canceled tick 2024-03-25T18:35:54.895 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.893+0000 7ff27448d640 1 -- 172.21.15.99:0/1394065122 >> v1:172.21.15.190:6800/4220409699 conn(0x7ff25006e0c0 legacy=0x7ff250070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:54.895 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.894+0000 7ff27448d640 10 monclient: shutdown 2024-03-25T18:35:54.895 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.894+0000 7ff27448d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:54.895 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.894+0000 7ff27448d640 1 -- 172.21.15.99:0/1394065122 >> v1:172.21.15.190:6789/0 conn(0x7ff26c0a98f0 legacy=0x7ff26c1db1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:54.895 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.894+0000 7ff27448d640 1 -- 172.21.15.99:0/1394065122 shutdown_connections 2024-03-25T18:35:54.895 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.894+0000 7ff27448d640 1 -- 172.21.15.99:0/1394065122 >> 172.21.15.99:0/1394065122 conn(0x7ff26c0b7ea0 msgr2=0x7ff26c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:54.895 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.894+0000 7ff27448d640 1 -- 172.21.15.99:0/1394065122 shutdown_connections 2024-03-25T18:35:54.896 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.894+0000 7ff27448d640 1 -- 172.21.15.99:0/1394065122 wait complete. 2024-03-25T18:35:54.896 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:35:54.894+0000 7ff27448d640 1 librados: shutdown 2024-03-25T18:35:54.906 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:55.381 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:10:55.973 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:55.388 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:10:55.981 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:55.427 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:10:56.020 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:55.443 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:10:56.036 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:55.450 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:10:56.043 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:55.456 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:10:56.015 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:55.461 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:10:56.020 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:55.475 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:10:56.068 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:55.483 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:10:56.042 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:01.714 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:01.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.825+0000 7f643aef6640 10 monclient: get_monmap_and_config 2024-03-25T18:36:01.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.825+0000 7f643aef6640 10 monclient: build_initial_monmap 2024-03-25T18:36:01.826 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.825+0000 7f643aef6640 10 monclient: monmap: 2024-03-25T18:36:01.826 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:01.826 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:01.826 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:36:01.826465+0000 2024-03-25T18:36:01.827 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:36:01.826465+0000 2024-03-25T18:36:01.827 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:01.827 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:01.827 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:01.827 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:36:01.827 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:36:01.827 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:01.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.826+0000 7f643aef6640 1 Processor -- start 2024-03-25T18:36:01.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.826+0000 7f643aef6640 1 -- start start 2024-03-25T18:36:01.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.826+0000 7f643aef6640 10 monclient: init 2024-03-25T18:36:01.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.827+0000 7f643aef6640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:01.828 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.827+0000 7f643aef6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:01.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.827+0000 7f643aef6640 10 monclient(hunting): picked mon.noname-c con 0x7f64340a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:01.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.827+0000 7f643aef6640 10 monclient(hunting): picked mon.noname-b con 0x7f643413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:01.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.827+0000 7f643aef6640 10 monclient(hunting): picked mon.noname-a con 0x7f64341430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:01.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.827+0000 7f643aef6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6434142520 con 0x7f64341430c0 2024-03-25T18:36:01.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.827+0000 7f643aef6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64340b28a0 con 0x7f64340a98f0 2024-03-25T18:36:01.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.827+0000 7f643aef6640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64340b2a30 con 0x7f643413f710 2024-03-25T18:36:01.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.827+0000 7f643aef6640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:01.829 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.827+0000 7f643aef6640 10 monclient(hunting): authenticate will time out at 7979.995039s 2024-03-25T18:36:01.836 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.835+0000 7f6438c92640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f64340a98f0 0x7f64340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:51776/0 (socket says 172.21.15.99:51776) 2024-03-25T18:36:01.836 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.835+0000 7f6438c92640 1 -- 172.21.15.99:0/569216870 learned_addr learned my addr 172.21.15.99:0/569216870 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:01.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.905+0000 7f642b7fe640 1 -- 172.21.15.99:0/569216870 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590468267 0 0) 0x7f6434142520 con 0x7f64341430c0 2024-03-25T18:36:01.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.905+0000 7f642b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:01.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.905+0000 7f642b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:01.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.905+0000 7f642b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:01.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.905+0000 7f642b7fe640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:36:01.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.905+0000 7f642b7fe640 1 -- 172.21.15.99:0/569216870 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f641c003520 con 0x7f64341430c0 2024-03-25T18:36:01.926 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 1 -- 172.21.15.99:0/569216870 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 293141120 0 0) 0x7f641c003520 con 0x7f64341430c0 2024-03-25T18:36:01.926 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 1 -- 172.21.15.99:0/569216870 >> v1:172.21.15.99:6790/0 conn(0x7f643413f710 legacy=0x7f64341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:01.926 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 1 -- 172.21.15.99:0/569216870 >> v1:172.21.15.190:6789/0 conn(0x7f64340a98f0 legacy=0x7f64340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:01.926 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:36:01.926 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:36:01.926 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:01.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:36:01.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 1 -- 172.21.15.99:0/569216870 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64340b44c0 con 0x7f64341430c0 2024-03-25T18:36:01.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:36:01.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:36:01.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:01.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 1 -- 172.21.15.99:0/569216870 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f642c001a10 con 0x7f64341430c0 2024-03-25T18:36:01.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f643aef6640 5 monclient: authenticate success, global_id 4695 2024-03-25T18:36:01.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f643aef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:01.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:01.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:01.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f642b7fe640 10 monclient: dump: 2024-03-25T18:36:01.927 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:01.927 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:01.927 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:01.928 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:01.928 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:01.928 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:01.928 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:01.928 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:01.928 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:01.928 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:01.928 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.925+0000 7f643aef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:01.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.934+0000 7f642b7fe640 1 -- 172.21.15.99:0/569216870 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f642c001b70 con 0x7f64341430c0 2024-03-25T18:36:01.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.934+0000 7f642b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:01.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.934+0000 7f642b7fe640 1 -- 172.21.15.99:0/569216870 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f642c001f80 con 0x7f64341430c0 2024-03-25T18:36:01.942 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.934+0000 7f642b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:01.942 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.934+0000 7f642b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:01.942 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.934+0000 7f642b7fe640 10 monclient: dump: 2024-03-25T18:36:01.942 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:01.942 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:01.942 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:01.942 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:01.942 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:01.942 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:01.942 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:01.942 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:01.942 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:01.942 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:01.942 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.934+0000 7f643aef6640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.934+0000 7f643aef6640 4 set_mon_vals no callback set 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.935+0000 7f643aef6640 10 monclient: shutdown 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.935+0000 7f643aef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.935+0000 7f643aef6640 1 -- 172.21.15.99:0/569216870 >> v1:172.21.15.99:6789/0 conn(0x7f64341430c0 legacy=0x7f643414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.935+0000 7f643aef6640 1 -- 172.21.15.99:0/569216870 shutdown_connections 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.935+0000 7f643aef6640 1 -- 172.21.15.99:0/569216870 >> 172.21.15.99:0/569216870 conn(0x7f64340b7ea0 msgr2=0x7f64340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.935+0000 7f643aef6640 1 -- 172.21.15.99:0/569216870 shutdown_connections 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.935+0000 7f643aef6640 1 -- 172.21.15.99:0/569216870 wait complete. 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.936+0000 7f643aef6640 10 monclient: build_initial_monmap 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.936+0000 7f643aef6640 10 monclient: monmap: 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:36:01.943 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.937+0000 7f643aef6640 1 librados: starting msgr at 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.937+0000 7f643aef6640 1 librados: starting objecter 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.937+0000 7f643aef6640 1 Processor -- start 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.937+0000 7f643aef6640 1 -- start start 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.937+0000 7f643aef6640 1 librados: setting wanted keys 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.937+0000 7f643aef6640 1 librados: calling monclient init 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.937+0000 7f643aef6640 10 monclient: init 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.938+0000 7f643aef6640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.938+0000 7f643aef6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.938+0000 7f643aef6640 10 monclient(hunting): picked mon.noname-a con 0x7f64340a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:01.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.938+0000 7f643aef6640 10 monclient(hunting): picked mon.noname-c con 0x7f643413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:01.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.938+0000 7f643aef6640 10 monclient(hunting): picked mon.noname-b con 0x7f64341430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:01.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.938+0000 7f643aef6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643406fbe0 con 0x7f64340a98f0 2024-03-25T18:36:01.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.938+0000 7f643aef6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643406fec0 con 0x7f64341430c0 2024-03-25T18:36:01.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.938+0000 7f643aef6640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643406eeb0 con 0x7f643413f710 2024-03-25T18:36:01.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.938+0000 7f643aef6640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:01.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.938+0000 7f643aef6640 10 monclient(hunting): authenticate will time out at 7980.106267s 2024-03-25T18:36:01.948 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.947+0000 7f6438c92640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f64340a98f0 0x7f643414bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:46896/0 (socket says 172.21.15.99:46896) 2024-03-25T18:36:01.948 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.947+0000 7f6438c92640 1 -- 172.21.15.99:0/2027198152 learned_addr learned my addr 172.21.15.99:0/2027198152 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:01.953 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.952+0000 7f64297fa640 10 client.?.objecter ms_handle_connect 0x7f64340a98f0 2024-03-25T18:36:01.953 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.952+0000 7f64297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:01.964 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.964+0000 7f64297fa640 10 client.?.objecter ms_handle_connect 0x7f64341430c0 2024-03-25T18:36:01.965 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.964+0000 7f64297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:01.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.965+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456815205 0 0) 0x7f643406fbe0 con 0x7f64340a98f0 2024-03-25T18:36:01.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.965+0000 7f64297fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:01.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.965+0000 7f64297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:01.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.965+0000 7f64297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:01.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.965+0000 7f64297fa640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:36:01.966 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.965+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6420004eb0 con 0x7f64340a98f0 2024-03-25T18:36:01.979 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.978+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1820380554 0 0) 0x7f6420004eb0 con 0x7f64340a98f0 2024-03-25T18:36:01.980 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.978+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f643406fbe0 con 0x7f64340a98f0 2024-03-25T18:36:01.980 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.978+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f64300018b0 con 0x7f64340a98f0 2024-03-25T18:36:01.980 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.978+0000 7f64297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:01.980 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.978+0000 7f64297fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:01.980 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.978+0000 7f64297fa640 10 monclient(hunting): dump: 2024-03-25T18:36:01.980 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:01.980 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:01.980 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:01.980 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:01.980 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:01.980 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:01.980 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:01.980 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:01.980 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:01.981 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:01.981 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.978+0000 7f64297fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:01.981 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.979+0000 7f64297fa640 10 client.?.objecter ms_handle_connect 0x7f643413f710 2024-03-25T18:36:01.981 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.979+0000 7f64297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:01.987 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.986+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120241549 0 0) 0x7f643406fec0 con 0x7f64341430c0 2024-03-25T18:36:01.987 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.986+0000 7f64297fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:01.987 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.986+0000 7f64297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:01.987 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.986+0000 7f64297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:01.987 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.986+0000 7f64297fa640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:36:01.987 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.986+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6420004eb0 con 0x7f64341430c0 2024-03-25T18:36:01.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 133570727 0 0) 0x7f643406fbe0 con 0x7f64340a98f0 2024-03-25T18:36:01.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 >> v1:172.21.15.99:6790/0 conn(0x7f643413f710 legacy=0x7f643414c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:01.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 >> v1:172.21.15.190:6789/0 conn(0x7f64341430c0 legacy=0x7f64341e62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:01.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f64297fa640 10 monclient: _finish_hunting 0 2024-03-25T18:36:01.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f64297fa640 1 monclient: found mon.a 2024-03-25T18:36:01.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f64297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:01.994 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f64297fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643414e6c0 con 0x7f64340a98f0 2024-03-25T18:36:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f64297fa640 10 monclient: _finish_auth 0 2024-03-25T18:36:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f64297fa640 10 monclient: _check_auth_tickets 2024-03-25T18:36:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f64297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f643aef6640 5 monclient: authenticate success, global_id 4698 2024-03-25T18:36:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f643aef6640 10 monclient: _renew_subs 2024-03-25T18:36:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f643aef6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.992+0000 7f643aef6640 1 -- 172.21.15.99:0/2027198152 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6434070930 con 0x7f64340a98f0 2024-03-25T18:36:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.993+0000 7f643aef6640 10 client.4698.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.993+0000 7f643aef6640 10 monclient: _renew_subs 2024-03-25T18:36:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.993+0000 7f643aef6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.993+0000 7f643aef6640 1 -- 172.21.15.99:0/2027198152 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6434070cd0 con 0x7f64340a98f0 2024-03-25T18:36:01.995 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.993+0000 7f643aef6640 1 librados: init done 2024-03-25T18:36:01.996 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.994+0000 7f643aef6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:01.996 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.994+0000 7f643aef6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:01.996 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.994+0000 7f643aef6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:01.996 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:01.994+0000 7f643aef6640 1 -- 172.21.15.99:0/2027198152 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6434071370 con 0x7f64340a98f0 2024-03-25T18:36:02.301 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.300+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64300047e0 con 0x7f64340a98f0 2024-03-25T18:36:02.301 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.300+0000 7f64297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:02.301 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.300+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6430004b70 con 0x7f64340a98f0 2024-03-25T18:36:02.301 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.300+0000 7f64297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:02.301 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.300+0000 7f64297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:02.301 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.300+0000 7f64297fa640 10 monclient: dump: 2024-03-25T18:36:02.301 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:02.301 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:02.301 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:02.301 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:02.301 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:02.302 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:02.302 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:02.302 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:02.302 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:02.302 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:02.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.300+0000 7f642affd640 4 set_mon_vals no callback set 2024-03-25T18:36:02.302 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.301+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f6430004d90 con 0x7f64340a98f0 2024-03-25T18:36:02.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.301+0000 7f642bfff640 10 monclient: get_auth_request con 0x7f642006dd40 auth_method 0 2024-03-25T18:36:02.303 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.302+0000 7f64297fa640 10 client.4698.objecter ms_handle_connect 0x7f642006dd40 2024-03-25T18:36:02.309 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.308+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2833420883 0 0) 0x7f64300548f0 con 0x7f64340a98f0 2024-03-25T18:36:02.309 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.308+0000 7f64297fa640 10 client.4698.objecter ms_dispatch 0x7f6434000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:02.309 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.308+0000 7f64297fa640 3 client.4698.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:02.309 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.308+0000 7f64297fa640 3 client.4698.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:02.309 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.308+0000 7f64297fa640 20 client.4698.objecter dump_active .. 0 homeless 2024-03-25T18:36:02.309 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.308+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6430050400 con 0x7f64340a98f0 2024-03-25T18:36:02.309 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.308+0000 7f64297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:02.310 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.308+0000 7f64297fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:02.562 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.561+0000 7f643aef6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:02.562 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.561+0000 7f643aef6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:02.562 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.561+0000 7f643aef6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:02.562 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.561+0000 7f643aef6640 1 -- 172.21.15.99:0/2027198152 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6434070ac0 con 0x7f64340a98f0 2024-03-25T18:36:02.589 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.588+0000 7f64297fa640 1 -- 172.21.15.99:0/2027198152 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 270404165) 0x7f6430052330 con 0x7f64340a98f0 2024-03-25T18:36:02.589 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.588+0000 7f64297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:02.589 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.588+0000 7f64297fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:02.589 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:36:02.590 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":11,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:35:20.161948+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:20.161948+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:02.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.591+0000 7f643aef6640 10 librados: watch_flush enter 2024-03-25T18:36:02.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.591+0000 7f643aef6640 10 librados: watch_flush exit 2024-03-25T18:36:02.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.591+0000 7f643aef6640 20 client.4698.objecter shutdown clearing up homeless session... 2024-03-25T18:36:02.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.591+0000 7f643aef6640 10 client.4698.objecter successfully canceled tick 2024-03-25T18:36:02.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.591+0000 7f643aef6640 1 -- 172.21.15.99:0/2027198152 >> v1:172.21.15.190:6800/4220409699 conn(0x7f642006dd40 legacy=0x7f6420070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:02.592 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.591+0000 7f643aef6640 10 monclient: shutdown 2024-03-25T18:36:02.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.591+0000 7f643aef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:02.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.591+0000 7f643aef6640 1 -- 172.21.15.99:0/2027198152 >> v1:172.21.15.99:6789/0 conn(0x7f64340a98f0 legacy=0x7f643414bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:02.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.591+0000 7f643aef6640 1 -- 172.21.15.99:0/2027198152 shutdown_connections 2024-03-25T18:36:02.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.591+0000 7f643aef6640 1 -- 172.21.15.99:0/2027198152 >> 172.21.15.99:0/2027198152 conn(0x7f64340b7ea0 msgr2=0x7f6434074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:02.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.591+0000 7f643aef6640 1 -- 172.21.15.99:0/2027198152 shutdown_connections 2024-03-25T18:36:02.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.591+0000 7f643aef6640 1 -- 172.21.15.99:0/2027198152 wait complete. 2024-03-25T18:36:02.593 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:02.591+0000 7f643aef6640 1 librados: shutdown 2024-03-25T18:36:02.603 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:09.410 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:09.524 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.522+0000 7f2404030640 10 monclient: get_monmap_and_config 2024-03-25T18:36:09.524 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.522+0000 7f2404030640 10 monclient: build_initial_monmap 2024-03-25T18:36:09.524 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.522+0000 7f2404030640 10 monclient: monmap: 2024-03-25T18:36:09.524 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:09.524 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:09.524 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:36:09.523981+0000 2024-03-25T18:36:09.524 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:36:09.523981+0000 2024-03-25T18:36:09.524 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:09.524 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:09.524 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:09.524 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:36:09.525 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:36:09.525 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:09.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.523+0000 7f2404030640 1 Processor -- start 2024-03-25T18:36:09.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.524+0000 7f2404030640 1 -- start start 2024-03-25T18:36:09.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.524+0000 7f2404030640 10 monclient: init 2024-03-25T18:36:09.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.525+0000 7f2404030640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:09.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.525+0000 7f2404030640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:09.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.525+0000 7f2404030640 10 monclient(hunting): picked mon.noname-b con 0x7f23fc0a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:09.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.525+0000 7f2404030640 10 monclient(hunting): picked mon.noname-c con 0x7f23fc13f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:09.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.525+0000 7f2404030640 10 monclient(hunting): picked mon.noname-a con 0x7f23fc1430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:09.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.525+0000 7f2404030640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc142520 con 0x7f23fc1430c0 2024-03-25T18:36:09.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.525+0000 7f2404030640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc0b28a0 con 0x7f23fc13f710 2024-03-25T18:36:09.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.525+0000 7f2404030640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc0b2a30 con 0x7f23fc0a98f0 2024-03-25T18:36:09.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.525+0000 7f2404030640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:09.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.525+0000 7f2404030640 10 monclient(hunting): authenticate will time out at 7987.693048s 2024-03-25T18:36:09.549 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.549+0000 7f24015cb640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f23fc13f710 0x7f23fc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:37416/0 (socket says 172.21.15.99:37416) 2024-03-25T18:36:09.550 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.549+0000 7f24015cb640 1 -- 172.21.15.99:0/3733614442 learned_addr learned my addr 172.21.15.99:0/3733614442 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:09.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.568+0000 7f2400dca640 1 -- 172.21.15.99:0/3733614442 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168366691 0 0) 0x7f23fc142520 con 0x7f23fc1430c0 2024-03-25T18:36:09.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.568+0000 7f2400dca640 10 monclient(hunting): _negotiate 2024-03-25T18:36:09.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.568+0000 7f2400dca640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:09.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.568+0000 7f2400dca640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:09.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.568+0000 7f2400dca640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:36:09.569 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.568+0000 7f2400dca640 1 -- 172.21.15.99:0/3733614442 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23e8003520 con 0x7f23fc1430c0 2024-03-25T18:36:09.576 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.576+0000 7f2400dca640 1 -- 172.21.15.99:0/3733614442 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436964750 0 0) 0x7f23fc0b28a0 con 0x7f23fc13f710 2024-03-25T18:36:09.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.576+0000 7f2400dca640 10 monclient(hunting): _negotiate 2024-03-25T18:36:09.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.576+0000 7f2400dca640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:09.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.576+0000 7f2400dca640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:09.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.576+0000 7f2400dca640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:36:09.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.576+0000 7f2400dca640 1 -- 172.21.15.99:0/3733614442 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23fc142520 con 0x7f23fc13f710 2024-03-25T18:36:09.577 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.576+0000 7f2400dca640 1 -- 172.21.15.99:0/3733614442 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439384982 0 0) 0x7f23fc0b2a30 con 0x7f23fc0a98f0 2024-03-25T18:36:09.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.576+0000 7f2400dca640 10 monclient(hunting): _negotiate 2024-03-25T18:36:09.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.576+0000 7f2400dca640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:09.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.576+0000 7f2400dca640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:09.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.576+0000 7f2400dca640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:36:09.578 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.576+0000 7f2400dca640 1 -- 172.21.15.99:0/3733614442 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23fc0b28a0 con 0x7f23fc0a98f0 2024-03-25T18:36:09.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 1 -- 172.21.15.99:0/3733614442 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3562818797 0 0) 0x7f23e8003520 con 0x7f23fc1430c0 2024-03-25T18:36:09.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 1 -- 172.21.15.99:0/3733614442 >> v1:172.21.15.99:6790/0 conn(0x7f23fc0a98f0 legacy=0x7f23fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:09.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 1 -- 172.21.15.99:0/3733614442 >> v1:172.21.15.190:6789/0 conn(0x7f23fc13f710 legacy=0x7f23fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:09.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 10 monclient: _finish_hunting 0 2024-03-25T18:36:09.579 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 1 monclient: found mon.noname-a 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 1 -- 172.21.15.99:0/3733614442 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc0b44c0 con 0x7f23fc1430c0 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 10 monclient: _finish_auth 0 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 10 monclient: _check_auth_tickets 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 1 -- 172.21.15.99:0/3733614442 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f23f4001a10 con 0x7f23fc1430c0 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2400dca640 10 monclient: dump: 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:09.580 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:09.581 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:09.581 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:09.581 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:09.581 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:09.581 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:09.581 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2404030640 5 monclient: authenticate success, global_id 4701 2024-03-25T18:36:09.581 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.578+0000 7f2404030640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:09.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.582+0000 7f2400dca640 1 -- 172.21.15.99:0/3733614442 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23f4001b70 con 0x7f23fc1430c0 2024-03-25T18:36:09.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.582+0000 7f2400dca640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:09.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.582+0000 7f2404030640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:09.583 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.582+0000 7f2404030640 4 set_mon_vals no callback set 2024-03-25T18:36:09.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.583+0000 7f2404030640 10 monclient: shutdown 2024-03-25T18:36:09.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.583+0000 7f2404030640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:09.584 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.583+0000 7f2404030640 1 -- 172.21.15.99:0/3733614442 >> v1:172.21.15.99:6789/0 conn(0x7f23fc1430c0 legacy=0x7f23fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:09.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.583+0000 7f2404030640 1 -- 172.21.15.99:0/3733614442 shutdown_connections 2024-03-25T18:36:09.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.583+0000 7f2404030640 1 -- 172.21.15.99:0/3733614442 >> 172.21.15.99:0/3733614442 conn(0x7f23fc0b7ea0 msgr2=0x7f23fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:09.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.583+0000 7f2404030640 1 -- 172.21.15.99:0/3733614442 shutdown_connections 2024-03-25T18:36:09.585 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.583+0000 7f2404030640 1 -- 172.21.15.99:0/3733614442 wait complete. 2024-03-25T18:36:09.586 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.584+0000 7f2404030640 10 monclient: build_initial_monmap 2024-03-25T18:36:09.586 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.584+0000 7f2404030640 10 monclient: monmap: 2024-03-25T18:36:09.586 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:09.586 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.584+0000 7f2404030640 1 librados: starting msgr at 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.584+0000 7f2404030640 1 librados: starting objecter 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.584+0000 7f2404030640 1 Processor -- start 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.584+0000 7f2404030640 1 -- start start 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.584+0000 7f2404030640 1 librados: setting wanted keys 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.584+0000 7f2404030640 1 librados: calling monclient init 2024-03-25T18:36:09.587 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.584+0000 7f2404030640 10 monclient: init 2024-03-25T18:36:09.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.585+0000 7f2404030640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:09.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.585+0000 7f2404030640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:36:09.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.585+0000 7f2404030640 10 monclient(hunting): picked mon.noname-b con 0x7f23fc0a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:09.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.585+0000 7f2404030640 10 monclient(hunting): picked mon.noname-a con 0x7f23fc13f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:09.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.585+0000 7f2404030640 10 monclient(hunting): picked mon.noname-c con 0x7f23fc1430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:09.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.585+0000 7f2404030640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc1db7c0 con 0x7f23fc13f710 2024-03-25T18:36:09.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.585+0000 7f2404030640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc1dab90 con 0x7f23fc0a98f0 2024-03-25T18:36:09.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.585+0000 7f2404030640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23fc1398b0 con 0x7f23fc1430c0 2024-03-25T18:36:09.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.585+0000 7f2404030640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:09.588 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.585+0000 7f2404030640 10 monclient(hunting): authenticate will time out at 7987.753719s 2024-03-25T18:36:09.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.593+0000 7f2401dcc640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f23fc0a98f0 0x7f23fc1db0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:37460/0 (socket says 172.21.15.99:37460) 2024-03-25T18:36:09.594 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.593+0000 7f2401dcc640 1 -- 172.21.15.99:0/761490458 learned_addr learned my addr 172.21.15.99:0/761490458 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:09.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.594+0000 7f23f27fc640 10 client.?.objecter ms_handle_connect 0x7f23fc0a98f0 2024-03-25T18:36:09.595 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.594+0000 7f23f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:09.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.604+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007420347 0 0) 0x7f23fc1dab90 con 0x7f23fc0a98f0 2024-03-25T18:36:09.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.604+0000 7f23f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:09.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.604+0000 7f23f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:09.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.604+0000 7f23f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:09.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.604+0000 7f23f27fc640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:36:09.605 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.604+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23e0004eb0 con 0x7f23fc0a98f0 2024-03-25T18:36:09.611 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.611+0000 7f23f27fc640 10 client.?.objecter ms_handle_connect 0x7f23fc1430c0 2024-03-25T18:36:09.612 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.611+0000 7f23f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:09.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.618+0000 7f23f27fc640 10 client.?.objecter ms_handle_connect 0x7f23fc13f710 2024-03-25T18:36:09.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.618+0000 7f23f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:09.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.619+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448376317 0 0) 0x7f23fc1398b0 con 0x7f23fc1430c0 2024-03-25T18:36:09.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.619+0000 7f23f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:09.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.619+0000 7f23f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:09.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.619+0000 7f23f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:09.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.619+0000 7f23f27fc640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:36:09.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.619+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23fc1dab90 con 0x7f23fc1430c0 2024-03-25T18:36:09.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.623+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 317257621 0 0) 0x7f23e0004eb0 con 0x7f23fc0a98f0 2024-03-25T18:36:09.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.623+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23fc1398b0 con 0x7f23fc0a98f0 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.623+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f23f8001a10 con 0x7f23fc0a98f0 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.623+0000 7f23f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.624+0000 7f23f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.624+0000 7f23f27fc640 10 monclient(hunting): dump: 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:09.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.624+0000 7f23f27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:09.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2238376756 0 0) 0x7f23fc1398b0 con 0x7f23fc0a98f0 2024-03-25T18:36:09.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 >> v1:172.21.15.99:6790/0 conn(0x7f23fc1430c0 legacy=0x7f23fc1e2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:09.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 >> v1:172.21.15.99:6789/0 conn(0x7f23fc13f710 legacy=0x7f23fc133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:09.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f23f27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:36:09.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f23f27fc640 1 monclient: found mon.b 2024-03-25T18:36:09.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f23f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:09.631 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f23f27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:09.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc139a40 con 0x7f23fc0a98f0 2024-03-25T18:36:09.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f23f27fc640 10 monclient: _finish_auth 0 2024-03-25T18:36:09.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f23f27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:36:09.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f23f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:09.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f2404030640 5 monclient: authenticate success, global_id 4714 2024-03-25T18:36:09.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f2404030640 10 monclient: _renew_subs 2024-03-25T18:36:09.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f2404030640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:09.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f2404030640 1 -- 172.21.15.99:0/761490458 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23fc135490 con 0x7f23fc0a98f0 2024-03-25T18:36:09.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f2404030640 10 client.4714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:09.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f2404030640 10 monclient: _renew_subs 2024-03-25T18:36:09.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f2404030640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:09.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f2404030640 1 -- 172.21.15.99:0/761490458 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23fc1380e0 con 0x7f23fc0a98f0 2024-03-25T18:36:09.632 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.630+0000 7f2404030640 1 librados: init done 2024-03-25T18:36:09.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.631+0000 7f2404030640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:09.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.631+0000 7f2404030640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:09.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.631+0000 7f2404030640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:09.633 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.631+0000 7f2404030640 1 -- 172.21.15.99:0/761490458 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23fc0ac310 con 0x7f23fc0a98f0 2024-03-25T18:36:09.904 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.903+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23f8001c70 con 0x7f23fc0a98f0 2024-03-25T18:36:09.904 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.903+0000 7f23f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:09.904 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.903+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f23f8003a60 con 0x7f23fc0a98f0 2024-03-25T18:36:09.904 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.903+0000 7f23f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:09.904 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.903+0000 7f23f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:09.904 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.903+0000 7f23f27fc640 10 monclient: dump: 2024-03-25T18:36:09.904 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:09.904 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:09.905 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:09.905 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:09.905 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:09.905 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:09.905 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:09.905 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:09.905 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:09.905 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:09.905 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.903+0000 7f23f3fff640 4 set_mon_vals no callback set 2024-03-25T18:36:09.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.903+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f23f8003c80 con 0x7f23fc0a98f0 2024-03-25T18:36:09.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.904+0000 7f24015cb640 10 monclient: get_auth_request con 0x7f23e006dd10 auth_method 0 2024-03-25T18:36:09.906 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.904+0000 7f23f27fc640 10 client.4714.objecter ms_handle_connect 0x7f23e006dd10 2024-03-25T18:36:09.909 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.908+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2833420883 0 0) 0x7f23f8054830 con 0x7f23fc0a98f0 2024-03-25T18:36:09.909 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.908+0000 7f23f27fc640 10 client.4714.objecter ms_dispatch 0x7f23fc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:09.909 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.908+0000 7f23f27fc640 3 client.4714.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:09.909 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.908+0000 7f23f27fc640 3 client.4714.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:09.909 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.908+0000 7f23f27fc640 20 client.4714.objecter dump_active .. 0 homeless 2024-03-25T18:36:09.910 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.909+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23f8050340 con 0x7f23fc0a98f0 2024-03-25T18:36:09.910 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.909+0000 7f23f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:09.910 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:09.909+0000 7f23f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:10.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.166+0000 7f2404030640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:10.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.166+0000 7f2404030640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:10.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.166+0000 7f2404030640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:10.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.166+0000 7f2404030640 1 -- 172.21.15.99:0/761490458 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23fc1dad20 con 0x7f23fc0a98f0 2024-03-25T18:36:10.187 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.186+0000 7f23f27fc640 1 -- 172.21.15.99:0/761490458 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 270404165) 0x7f23f8052270 con 0x7f23fc0a98f0 2024-03-25T18:36:10.187 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.186+0000 7f23f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:10.187 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.186+0000 7f23f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:10.188 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:36:10.188 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":11,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:35:20.161948+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:20.161948+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:10.190 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.189+0000 7f2404030640 10 librados: watch_flush enter 2024-03-25T18:36:10.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.189+0000 7f2404030640 10 librados: watch_flush exit 2024-03-25T18:36:10.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.189+0000 7f2404030640 20 client.4714.objecter shutdown clearing up homeless session... 2024-03-25T18:36:10.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.189+0000 7f2404030640 10 client.4714.objecter successfully canceled tick 2024-03-25T18:36:10.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.189+0000 7f2404030640 1 -- 172.21.15.99:0/761490458 >> v1:172.21.15.190:6800/4220409699 conn(0x7f23e006dd10 legacy=0x7f23e00701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:10.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.189+0000 7f2404030640 10 monclient: shutdown 2024-03-25T18:36:10.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.189+0000 7f2404030640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:10.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.189+0000 7f2404030640 1 -- 172.21.15.99:0/761490458 >> v1:172.21.15.190:6789/0 conn(0x7f23fc0a98f0 legacy=0x7f23fc1db0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:10.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.189+0000 7f2404030640 1 -- 172.21.15.99:0/761490458 shutdown_connections 2024-03-25T18:36:10.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.189+0000 7f2404030640 1 -- 172.21.15.99:0/761490458 >> 172.21.15.99:0/761490458 conn(0x7f23fc0b7ea0 msgr2=0x7f23fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:10.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.189+0000 7f2404030640 1 -- 172.21.15.99:0/761490458 shutdown_connections 2024-03-25T18:36:10.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.189+0000 7f2404030640 1 -- 172.21.15.99:0/761490458 wait complete. 2024-03-25T18:36:10.191 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:10.189+0000 7f2404030640 1 librados: shutdown 2024-03-25T18:36:10.202 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:15.373 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:11:15.965 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:15.434 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:11:16.027 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:15.459 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:11:16.052 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:15.480 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:11:16.073 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:15.493 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:11:16.086 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:15.510 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:11:16.069 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:15.517 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:11:16.076 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:15.531 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:11:16.123 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:15.544 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:11:16.103 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:17.008 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:17.124 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.124+0000 7f9090eba640 10 monclient: get_monmap_and_config 2024-03-25T18:36:17.125 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.124+0000 7f9090eba640 10 monclient: build_initial_monmap 2024-03-25T18:36:17.125 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.124+0000 7f9090eba640 10 monclient: monmap: 2024-03-25T18:36:17.125 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:17.125 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:17.125 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:36:17.125163+0000 2024-03-25T18:36:17.125 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:36:17.125163+0000 2024-03-25T18:36:17.125 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:17.125 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:17.125 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:17.126 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:36:17.126 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:36:17.126 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:17.127 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.124+0000 7f9090eba640 1 Processor -- start 2024-03-25T18:36:17.127 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.124+0000 7f9090eba640 1 -- start start 2024-03-25T18:36:17.127 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.124+0000 7f9090eba640 10 monclient: init 2024-03-25T18:36:17.127 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.125+0000 7f9090eba640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:17.127 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.125+0000 7f9090eba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:36:17.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.125+0000 7f9090eba640 10 monclient(hunting): picked mon.noname-c con 0x7f908c0a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:17.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.125+0000 7f9090eba640 10 monclient(hunting): picked mon.noname-a con 0x7f908c13f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:17.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.125+0000 7f9090eba640 10 monclient(hunting): picked mon.noname-b con 0x7f908c1430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:17.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.125+0000 7f9090eba640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c142520 con 0x7f908c13f710 2024-03-25T18:36:17.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.125+0000 7f9090eba640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c0b28a0 con 0x7f908c0a98f0 2024-03-25T18:36:17.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.125+0000 7f9090eba640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c0b2a30 con 0x7f908c1430c0 2024-03-25T18:36:17.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.125+0000 7f9090eba640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:17.128 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.125+0000 7f9090eba640 10 monclient(hunting): authenticate will time out at 7995.293457s 2024-03-25T18:36:17.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.135+0000 7f9083fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f908c13f710 0x7f908c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:38530/0 (socket says 172.21.15.99:38530) 2024-03-25T18:36:17.136 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.135+0000 7f9083fff640 1 -- 172.21.15.99:0/2508455475 learned_addr learned my addr 172.21.15.99:0/2508455475 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:17.156 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.155+0000 7f9089d9b640 1 -- 172.21.15.99:0/2508455475 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694860626 0 0) 0x7f908c0b28a0 con 0x7f908c0a98f0 2024-03-25T18:36:17.156 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.155+0000 7f9089d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:36:17.157 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.155+0000 7f9089d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:17.157 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.155+0000 7f9089d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:17.157 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.155+0000 7f9089d9b640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:36:17.157 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.155+0000 7f9089d9b640 1 -- 172.21.15.99:0/2508455475 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9078003520 con 0x7f908c0a98f0 2024-03-25T18:36:17.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 1 -- 172.21.15.99:0/2508455475 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1815760047 0 0) 0x7f9078003520 con 0x7f908c0a98f0 2024-03-25T18:36:17.165 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 1 -- 172.21.15.99:0/2508455475 >> v1:172.21.15.99:6790/0 conn(0x7f908c1430c0 legacy=0x7f908c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:17.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 1 -- 172.21.15.99:0/2508455475 >> v1:172.21.15.99:6789/0 conn(0x7f908c13f710 legacy=0x7f908c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:17.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:36:17.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 1 monclient: found mon.noname-c 2024-03-25T18:36:17.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:17.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:36:17.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 1 -- 172.21.15.99:0/2508455475 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c0b44c0 con 0x7f908c0a98f0 2024-03-25T18:36:17.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 10 monclient: _finish_auth 0 2024-03-25T18:36:17.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:36:17.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:17.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 1 -- 172.21.15.99:0/2508455475 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f907c001690 con 0x7f908c0a98f0 2024-03-25T18:36:17.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:17.166 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:17.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9089d9b640 10 monclient: dump: 2024-03-25T18:36:17.167 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:17.167 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:17.167 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:17.167 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:17.167 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:17.167 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:17.167 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:17.167 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:17.167 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:17.167 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:17.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9090eba640 5 monclient: authenticate success, global_id 4717 2024-03-25T18:36:17.167 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.164+0000 7f9090eba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:17.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.172+0000 7f9089d9b640 1 -- 172.21.15.99:0/2508455475 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f907c0017f0 con 0x7f908c0a98f0 2024-03-25T18:36:17.173 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.172+0000 7f9089d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:17.174 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.172+0000 7f9089d9b640 1 -- 172.21.15.99:0/2508455475 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f907c001c40 con 0x7f908c0a98f0 2024-03-25T18:36:17.174 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.172+0000 7f9090eba640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:17.174 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.172+0000 7f9089d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:17.174 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.172+0000 7f9090eba640 4 set_mon_vals no callback set 2024-03-25T18:36:17.174 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.172+0000 7f9089d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:17.174 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.172+0000 7f9089d9b640 10 monclient: dump: 2024-03-25T18:36:17.174 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:17.174 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:17.174 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:17.174 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:17.174 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:17.174 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:17.175 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:17.175 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:17.175 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:17.175 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:17.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.173+0000 7f9090eba640 10 monclient: shutdown 2024-03-25T18:36:17.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.173+0000 7f9090eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:17.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.173+0000 7f9090eba640 1 -- 172.21.15.99:0/2508455475 >> v1:172.21.15.190:6789/0 conn(0x7f908c0a98f0 legacy=0x7f908c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:17.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.173+0000 7f9090eba640 1 -- 172.21.15.99:0/2508455475 shutdown_connections 2024-03-25T18:36:17.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.173+0000 7f9090eba640 1 -- 172.21.15.99:0/2508455475 >> 172.21.15.99:0/2508455475 conn(0x7f908c0b7ea0 msgr2=0x7f908c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:17.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.173+0000 7f9090eba640 1 -- 172.21.15.99:0/2508455475 shutdown_connections 2024-03-25T18:36:17.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.173+0000 7f9090eba640 1 -- 172.21.15.99:0/2508455475 wait complete. 2024-03-25T18:36:17.177 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.174+0000 7f9090eba640 10 monclient: build_initial_monmap 2024-03-25T18:36:17.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.174+0000 7f9090eba640 10 monclient: monmap: 2024-03-25T18:36:17.178 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:17.178 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:17.178 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:36:17.178 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:36:17.178 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:17.178 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:36:17.178 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:17.178 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:36:17.178 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:36:17.178 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:17.178 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.174+0000 7f9090eba640 1 librados: starting msgr at 2024-03-25T18:36:17.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.174+0000 7f9090eba640 1 librados: starting objecter 2024-03-25T18:36:17.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.174+0000 7f9090eba640 1 Processor -- start 2024-03-25T18:36:17.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.175+0000 7f9090eba640 1 -- start start 2024-03-25T18:36:17.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.175+0000 7f9090eba640 1 librados: setting wanted keys 2024-03-25T18:36:17.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.175+0000 7f9090eba640 1 librados: calling monclient init 2024-03-25T18:36:17.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.175+0000 7f9090eba640 10 monclient: init 2024-03-25T18:36:17.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.175+0000 7f9090eba640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:17.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.175+0000 7f9090eba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:36:17.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.175+0000 7f9090eba640 10 monclient(hunting): picked mon.noname-a con 0x7f908c0a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:17.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.175+0000 7f9090eba640 10 monclient(hunting): picked mon.noname-b con 0x7f908c13f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:17.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.175+0000 7f9090eba640 10 monclient(hunting): picked mon.noname-c con 0x7f908c1430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:17.179 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.175+0000 7f9090eba640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c1db7d0 con 0x7f908c0a98f0 2024-03-25T18:36:17.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.175+0000 7f9090eba640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c0b1420 con 0x7f908c13f710 2024-03-25T18:36:17.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.175+0000 7f9090eba640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c0b15b0 con 0x7f908c1430c0 2024-03-25T18:36:17.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.175+0000 7f9090eba640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:17.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.175+0000 7f9090eba640 10 monclient(hunting): authenticate will time out at 7995.343749s 2024-03-25T18:36:17.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.179+0000 7f9083fff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f908c13f710 0x7f908c0b01e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:58040/0 (socket says 172.21.15.99:58040) 2024-03-25T18:36:17.180 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.179+0000 7f9083fff640 1 -- 172.21.15.99:0/139995069 learned_addr learned my addr 172.21.15.99:0/139995069 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:17.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.182+0000 7f9082ffd640 10 client.?.objecter ms_handle_connect 0x7f908c0a98f0 2024-03-25T18:36:17.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.182+0000 7f9082ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:17.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.182+0000 7f9082ffd640 10 client.?.objecter ms_handle_connect 0x7f908c13f710 2024-03-25T18:36:17.183 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.182+0000 7f9082ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:17.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.187+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119326612 0 0) 0x7f908c0b1420 con 0x7f908c13f710 2024-03-25T18:36:17.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.187+0000 7f9082ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:17.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.187+0000 7f9082ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:17.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.187+0000 7f9082ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:17.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.187+0000 7f9082ffd640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:36:17.188 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.187+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f906c004eb0 con 0x7f908c13f710 2024-03-25T18:36:17.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.192+0000 7f9082ffd640 10 client.?.objecter ms_handle_connect 0x7f908c1430c0 2024-03-25T18:36:17.193 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.192+0000 7f9082ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:17.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.195+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411864814 0 0) 0x7f908c1db7d0 con 0x7f908c0a98f0 2024-03-25T18:36:17.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.195+0000 7f9082ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:17.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.195+0000 7f9082ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:17.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.195+0000 7f9082ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:17.196 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.195+0000 7f9082ffd640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:36:17.197 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.195+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f908c0b1420 con 0x7f908c0a98f0 2024-03-25T18:36:17.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.199+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488244099 0 0) 0x7f908c0b15b0 con 0x7f908c1430c0 2024-03-25T18:36:17.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.199+0000 7f9082ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:17.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.199+0000 7f9082ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:17.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.199+0000 7f9082ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:17.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.199+0000 7f9082ffd640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:36:17.200 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.199+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f908c1db7d0 con 0x7f908c1430c0 2024-03-25T18:36:17.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.202+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4226998343 0 0) 0x7f906c004eb0 con 0x7f908c13f710 2024-03-25T18:36:17.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.202+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f908c0b15b0 con 0x7f908c13f710 2024-03-25T18:36:17.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.202+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f90740018b0 con 0x7f908c13f710 2024-03-25T18:36:17.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.202+0000 7f9082ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:17.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.202+0000 7f9082ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:17.204 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.202+0000 7f9082ffd640 10 monclient(hunting): dump: 2024-03-25T18:36:17.204 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:17.204 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:17.204 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:17.204 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:17.205 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:17.205 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:17.205 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:17.205 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:17.205 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:17.205 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:17.205 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.202+0000 7f9082ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:17.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.204+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1711476697 0 0) 0x7f908c0b1420 con 0x7f908c0a98f0 2024-03-25T18:36:17.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.204+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f906c004eb0 con 0x7f908c0a98f0 2024-03-25T18:36:17.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.204+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f907c001b40 con 0x7f908c0a98f0 2024-03-25T18:36:17.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.204+0000 7f9082ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:17.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.204+0000 7f9082ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:17.206 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.204+0000 7f9082ffd640 10 monclient(hunting): dump: 2024-03-25T18:36:17.206 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:17.206 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:17.206 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:17.206 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:17.207 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:17.207 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:17.207 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:17.207 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:17.207 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:17.207 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:17.212 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.211+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1038876200 0 0) 0x7f908c1db7d0 con 0x7f908c1430c0 2024-03-25T18:36:17.212 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.211+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f908c0b1420 con 0x7f908c1430c0 2024-03-25T18:36:17.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.211+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9084001a90 con 0x7f908c1430c0 2024-03-25T18:36:17.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.211+0000 7f9082ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:17.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.211+0000 7f9082ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:17.213 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.211+0000 7f9082ffd640 10 monclient(hunting): dump: 2024-03-25T18:36:17.213 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:17.213 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:17.213 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:17.213 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:17.213 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:17.213 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:17.213 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:17.213 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:17.213 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:17.214 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:17.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 965501251 0 0) 0x7f908c0b15b0 con 0x7f908c13f710 2024-03-25T18:36:17.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 >> v1:172.21.15.99:6790/0 conn(0x7f908c1430c0 legacy=0x7f908c1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:17.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 >> v1:172.21.15.99:6789/0 conn(0x7f908c0a98f0 legacy=0x7f908c1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:17.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9082ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:36:17.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9082ffd640 1 monclient: found mon.b 2024-03-25T18:36:17.218 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9082ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:17.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9082ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:17.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c0b1740 con 0x7f908c13f710 2024-03-25T18:36:17.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9082ffd640 10 monclient: _finish_auth 0 2024-03-25T18:36:17.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9082ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:36:17.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9082ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:17.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9090eba640 5 monclient: authenticate success, global_id 4720 2024-03-25T18:36:17.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9090eba640 10 monclient: _renew_subs 2024-03-25T18:36:17.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9090eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:17.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9090eba640 1 -- 172.21.15.99:0/139995069 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908c1db1a0 con 0x7f908c13f710 2024-03-25T18:36:17.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9090eba640 10 client.4720.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:17.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9090eba640 10 monclient: _renew_subs 2024-03-25T18:36:17.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9090eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:17.219 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9090eba640 1 -- 172.21.15.99:0/139995069 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f908c0b08f0 con 0x7f908c13f710 2024-03-25T18:36:17.220 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.217+0000 7f9090eba640 1 librados: init done 2024-03-25T18:36:17.220 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.218+0000 7f9090eba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:17.220 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.218+0000 7f9090eba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:17.220 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.218+0000 7f9090eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:17.220 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.218+0000 7f9090eba640 1 -- 172.21.15.99:0/139995069 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908c0627b0 con 0x7f908c13f710 2024-03-25T18:36:17.501 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.500+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90740047e0 con 0x7f908c13f710 2024-03-25T18:36:17.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.500+0000 7f9082ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:17.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.500+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f9074004c30 con 0x7f908c13f710 2024-03-25T18:36:17.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.500+0000 7f9082ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:17.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.500+0000 7f9082ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:17.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.500+0000 7f9082ffd640 10 monclient: dump: 2024-03-25T18:36:17.502 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:17.502 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:17.502 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:17.502 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:17.502 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:17.503 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:17.503 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:17.503 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:17.503 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:17.503 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:17.503 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.500+0000 7f908959a640 4 set_mon_vals no callback set 2024-03-25T18:36:17.504 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.501+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f9074004e50 con 0x7f908c13f710 2024-03-25T18:36:17.504 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.502+0000 7f908a59c640 10 monclient: get_auth_request con 0x7f906c06e0f0 auth_method 0 2024-03-25T18:36:17.504 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.502+0000 7f9082ffd640 10 client.4720.objecter ms_handle_connect 0x7f906c06e0f0 2024-03-25T18:36:17.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.505+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2833420883 0 0) 0x7f90740547d0 con 0x7f908c13f710 2024-03-25T18:36:17.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.505+0000 7f9082ffd640 10 client.4720.objecter ms_dispatch 0x7f908c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:17.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.505+0000 7f9082ffd640 3 client.4720.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:17.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.505+0000 7f9082ffd640 3 client.4720.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:17.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.505+0000 7f9082ffd640 20 client.4720.objecter dump_active .. 0 homeless 2024-03-25T18:36:17.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.509+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90740502e0 con 0x7f908c13f710 2024-03-25T18:36:17.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.509+0000 7f9082ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:17.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.509+0000 7f9082ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:17.755 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.754+0000 7f9090eba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:17.755 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.754+0000 7f9090eba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:17.755 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.754+0000 7f9090eba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:17.756 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.754+0000 7f9090eba640 1 -- 172.21.15.99:0/139995069 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f908c0abae0 con 0x7f908c13f710 2024-03-25T18:36:17.883 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.882+0000 7f9082ffd640 1 -- 172.21.15.99:0/139995069 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 270404165) 0x7f9074052210 con 0x7f908c13f710 2024-03-25T18:36:17.883 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.882+0000 7f9082ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:17.883 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.882+0000 7f9082ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:17.883 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:36:17.883 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":11,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:35:20.161948+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:20.161948+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:17.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.884+0000 7f9090eba640 10 librados: watch_flush enter 2024-03-25T18:36:17.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.884+0000 7f9090eba640 10 librados: watch_flush exit 2024-03-25T18:36:17.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.884+0000 7f9090eba640 20 client.4720.objecter shutdown clearing up homeless session... 2024-03-25T18:36:17.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.884+0000 7f9090eba640 10 client.4720.objecter successfully canceled tick 2024-03-25T18:36:17.885 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.884+0000 7f9090eba640 1 -- 172.21.15.99:0/139995069 >> v1:172.21.15.190:6800/4220409699 conn(0x7f906c06e0f0 legacy=0x7f906c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:17.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.884+0000 7f9090eba640 10 monclient: shutdown 2024-03-25T18:36:17.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.884+0000 7f9090eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:17.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.884+0000 7f9090eba640 1 -- 172.21.15.99:0/139995069 >> v1:172.21.15.190:6789/0 conn(0x7f908c13f710 legacy=0x7f908c0b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:17.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.884+0000 7f9090eba640 1 -- 172.21.15.99:0/139995069 shutdown_connections 2024-03-25T18:36:17.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.884+0000 7f9090eba640 1 -- 172.21.15.99:0/139995069 >> 172.21.15.99:0/139995069 conn(0x7f908c0b7ea0 msgr2=0x7f908c13c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:17.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.884+0000 7f9090eba640 1 -- 172.21.15.99:0/139995069 shutdown_connections 2024-03-25T18:36:17.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.884+0000 7f9090eba640 1 -- 172.21.15.99:0/139995069 wait complete. 2024-03-25T18:36:17.886 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:17.884+0000 7f9090eba640 1 librados: shutdown 2024-03-25T18:36:17.899 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:24.706 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:24.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.816+0000 7fb68091c640 10 monclient: get_monmap_and_config 2024-03-25T18:36:24.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.816+0000 7fb68091c640 10 monclient: build_initial_monmap 2024-03-25T18:36:24.817 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.816+0000 7fb68091c640 10 monclient: monmap: 2024-03-25T18:36:24.817 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:24.817 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:24.817 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:36:24.817455+0000 2024-03-25T18:36:24.817 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:36:24.817455+0000 2024-03-25T18:36:24.817 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:24.818 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:24.818 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:24.818 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:36:24.818 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:36:24.818 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:24.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.817+0000 7fb68091c640 1 Processor -- start 2024-03-25T18:36:24.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.817+0000 7fb68091c640 1 -- start start 2024-03-25T18:36:24.819 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.817+0000 7fb68091c640 10 monclient: init 2024-03-25T18:36:24.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.817+0000 7fb68091c640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:24.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.817+0000 7fb68091c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:36:24.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.817+0000 7fb68091c640 10 monclient(hunting): picked mon.noname-b con 0x7fb67c0b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:24.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.818+0000 7fb68091c640 10 monclient(hunting): picked mon.noname-a con 0x7fb67c0b1730 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:24.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.818+0000 7fb68091c640 10 monclient(hunting): picked mon.noname-c con 0x7fb67c14be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:24.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.818+0000 7fb68091c640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c0af940 con 0x7fb67c0b1730 2024-03-25T18:36:24.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.818+0000 7fb68091c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c0b0a70 con 0x7fb67c14be90 2024-03-25T18:36:24.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.818+0000 7fb68091c640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c0b0d50 con 0x7fb67c0b4d60 2024-03-25T18:36:24.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.818+0000 7fb68091c640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:24.820 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.818+0000 7fb68091c640 10 monclient(hunting): authenticate will time out at 8002.986086s 2024-03-25T18:36:24.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.831+0000 7fb67a59c640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7fb67c0b4d60 0x7fb67c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:49246/0 (socket says 172.21.15.99:49246) 2024-03-25T18:36:24.832 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.831+0000 7fb67a59c640 1 -- 172.21.15.99:0/2414278699 learned_addr learned my addr 172.21.15.99:0/2414278699 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:24.845 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.844+0000 7fb67959a640 1 -- 172.21.15.99:0/2414278699 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008085019 0 0) 0x7fb67c0b0d50 con 0x7fb67c0b4d60 2024-03-25T18:36:24.845 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.844+0000 7fb67959a640 10 monclient(hunting): _negotiate 2024-03-25T18:36:24.845 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.844+0000 7fb67959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:24.845 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.844+0000 7fb67959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:24.845 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.844+0000 7fb67959a640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:36:24.845 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.844+0000 7fb67959a640 1 -- 172.21.15.99:0/2414278699 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb664003520 con 0x7fb67c0b4d60 2024-03-25T18:36:24.850 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.849+0000 7fb67959a640 1 -- 172.21.15.99:0/2414278699 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246736270 0 0) 0x7fb67c0af940 con 0x7fb67c0b1730 2024-03-25T18:36:24.850 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.849+0000 7fb67959a640 10 monclient(hunting): _negotiate 2024-03-25T18:36:24.850 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.849+0000 7fb67959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:24.850 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.849+0000 7fb67959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:24.850 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.849+0000 7fb67959a640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:36:24.850 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.849+0000 7fb67959a640 1 -- 172.21.15.99:0/2414278699 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb67c0b0d50 con 0x7fb67c0b1730 2024-03-25T18:36:24.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.849+0000 7fb67959a640 1 -- 172.21.15.99:0/2414278699 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172803327 0 0) 0x7fb67c0b0a70 con 0x7fb67c14be90 2024-03-25T18:36:24.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.849+0000 7fb67959a640 10 monclient(hunting): _negotiate 2024-03-25T18:36:24.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.849+0000 7fb67959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:24.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.849+0000 7fb67959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:24.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.849+0000 7fb67959a640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:36:24.851 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.849+0000 7fb67959a640 1 -- 172.21.15.99:0/2414278699 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb67c0af940 con 0x7fb67c14be90 2024-03-25T18:36:24.861 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 1 -- 172.21.15.99:0/2414278699 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4266546598 0 0) 0x7fb664003520 con 0x7fb67c0b4d60 2024-03-25T18:36:24.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 1 -- 172.21.15.99:0/2414278699 >> v1:172.21.15.190:6789/0 conn(0x7fb67c14be90 legacy=0x7fb67c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:24.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 1 -- 172.21.15.99:0/2414278699 >> v1:172.21.15.99:6789/0 conn(0x7fb67c0b1730 legacy=0x7fb67c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:24.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 10 monclient: _finish_hunting 0 2024-03-25T18:36:24.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 1 monclient: found mon.noname-b 2024-03-25T18:36:24.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:24.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.99:6790/0 2024-03-25T18:36:24.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 1 -- 172.21.15.99:0/2414278699 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c0b1030 con 0x7fb67c0b4d60 2024-03-25T18:36:24.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 10 monclient: _finish_auth 0 2024-03-25T18:36:24.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 10 monclient: _check_auth_tickets 2024-03-25T18:36:24.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:24.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 1 -- 172.21.15.99:0/2414278699 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb66c001690 con 0x7fb67c0b4d60 2024-03-25T18:36:24.862 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb68091c640 5 monclient: authenticate success, global_id 4721 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb68091c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb67959a640 10 monclient: dump: 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:24.863 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.860+0000 7fb68091c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:24.878 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.877+0000 7fb67959a640 1 -- 172.21.15.99:0/2414278699 <== mon.2 v1:172.21.15.99:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb66c0017f0 con 0x7fb67c0b4d60 2024-03-25T18:36:24.878 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.877+0000 7fb67959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:24.879 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.877+0000 7fb67959a640 1 -- 172.21.15.99:0/2414278699 <== mon.2 v1:172.21.15.99:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb66c001c00 con 0x7fb67c0b4d60 2024-03-25T18:36:24.879 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.877+0000 7fb67959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:24.879 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.877+0000 7fb67959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:24.879 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.877+0000 7fb67959a640 10 monclient: dump: 2024-03-25T18:36:24.879 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:24.879 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:24.879 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:24.879 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:24.879 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:24.879 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:24.879 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:24.879 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:24.879 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:24.879 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:24.880 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.878+0000 7fb68091c640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:24.880 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.878+0000 7fb68091c640 4 set_mon_vals no callback set 2024-03-25T18:36:24.881 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.878+0000 7fb68091c640 10 monclient: shutdown 2024-03-25T18:36:24.881 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.878+0000 7fb68091c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:24.881 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.878+0000 7fb68091c640 1 -- 172.21.15.99:0/2414278699 >> v1:172.21.15.99:6790/0 conn(0x7fb67c0b4d60 legacy=0x7fb67c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:24.881 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.879+0000 7fb68091c640 1 -- 172.21.15.99:0/2414278699 shutdown_connections 2024-03-25T18:36:24.881 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.879+0000 7fb68091c640 1 -- 172.21.15.99:0/2414278699 >> 172.21.15.99:0/2414278699 conn(0x7fb67c0617a0 msgr2=0x7fb67c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:24.881 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.879+0000 7fb68091c640 1 -- 172.21.15.99:0/2414278699 shutdown_connections 2024-03-25T18:36:24.881 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.879+0000 7fb68091c640 1 -- 172.21.15.99:0/2414278699 wait complete. 2024-03-25T18:36:24.881 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.880+0000 7fb68091c640 10 monclient: build_initial_monmap 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.880+0000 7fb68091c640 10 monclient: monmap: 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.880+0000 7fb68091c640 1 librados: starting msgr at 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.880+0000 7fb68091c640 1 librados: starting objecter 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.880+0000 7fb68091c640 1 Processor -- start 2024-03-25T18:36:24.882 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.880+0000 7fb68091c640 1 -- start start 2024-03-25T18:36:24.883 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.880+0000 7fb68091c640 1 librados: setting wanted keys 2024-03-25T18:36:24.883 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.880+0000 7fb68091c640 1 librados: calling monclient init 2024-03-25T18:36:24.883 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.880+0000 7fb68091c640 10 monclient: init 2024-03-25T18:36:24.883 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.881+0000 7fb68091c640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:24.883 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.881+0000 7fb68091c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:36:24.883 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.881+0000 7fb68091c640 10 monclient(hunting): picked mon.noname-a con 0x7fb67c0b1730 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:24.884 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.881+0000 7fb68091c640 10 monclient(hunting): picked mon.noname-c con 0x7fb67c0b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:24.884 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.881+0000 7fb68091c640 10 monclient(hunting): picked mon.noname-b con 0x7fb67c14be90 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:24.884 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.881+0000 7fb68091c640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c141bf0 con 0x7fb67c0b1730 2024-03-25T18:36:24.884 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.881+0000 7fb68091c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c141ed0 con 0x7fb67c14be90 2024-03-25T18:36:24.884 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.881+0000 7fb68091c640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c1421b0 con 0x7fb67c0b4d60 2024-03-25T18:36:24.884 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.881+0000 7fb68091c640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:24.884 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.881+0000 7fb68091c640 10 monclient(hunting): authenticate will time out at 8003.049037s 2024-03-25T18:36:24.899 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.898+0000 7fb67ad9d640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fb67c14be90 0x7fb67c1341f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:55264/0 (socket says 172.21.15.99:55264) 2024-03-25T18:36:24.899 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.898+0000 7fb67ad9d640 1 -- 172.21.15.99:0/1536073591 learned_addr learned my addr 172.21.15.99:0/1536073591 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:24.901 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.900+0000 7fb66affd640 10 client.?.objecter ms_handle_connect 0x7fb67c14be90 2024-03-25T18:36:24.901 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.900+0000 7fb66affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:24.905 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.904+0000 7fb66affd640 10 client.?.objecter ms_handle_connect 0x7fb67c0b1730 2024-03-25T18:36:24.905 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.904+0000 7fb66affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:24.910 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.909+0000 7fb66affd640 10 client.?.objecter ms_handle_connect 0x7fb67c0b4d60 2024-03-25T18:36:24.910 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.909+0000 7fb66affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:24.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.912+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695810442 0 0) 0x7fb67c141bf0 con 0x7fb67c0b1730 2024-03-25T18:36:24.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.912+0000 7fb66affd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:24.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.912+0000 7fb66affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:24.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.912+0000 7fb66affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:24.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.912+0000 7fb66affd640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:36:24.913 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.912+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb65c004eb0 con 0x7fb67c0b1730 2024-03-25T18:36:24.914 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.913+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193900116 0 0) 0x7fb67c141ed0 con 0x7fb67c14be90 2024-03-25T18:36:24.914 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.913+0000 7fb66affd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:24.914 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.913+0000 7fb66affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:24.914 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.913+0000 7fb66affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:24.914 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.913+0000 7fb66affd640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:36:24.914 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.913+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb67c141bf0 con 0x7fb67c14be90 2024-03-25T18:36:24.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.914+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776298995 0 0) 0x7fb67c1421b0 con 0x7fb67c0b4d60 2024-03-25T18:36:24.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.914+0000 7fb66affd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:24.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.914+0000 7fb66affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:24.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.914+0000 7fb66affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:24.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.914+0000 7fb66affd640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:36:24.915 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.914+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb67c141ed0 con 0x7fb67c0b4d60 2024-03-25T18:36:24.926 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.925+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 103835107 0 0) 0x7fb67c141ed0 con 0x7fb67c0b4d60 2024-03-25T18:36:24.926 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.925+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb67c1421b0 con 0x7fb67c0b4d60 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.925+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb670001930 con 0x7fb67c0b4d60 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.925+0000 7fb66affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.925+0000 7fb66affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.925+0000 7fb66affd640 10 monclient(hunting): dump: 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:24.927 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.925+0000 7fb66affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:24.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.940+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3680916096 0 0) 0x7fb65c004eb0 con 0x7fb67c0b1730 2024-03-25T18:36:24.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.940+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb67c141ed0 con 0x7fb67c0b1730 2024-03-25T18:36:24.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.940+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb66c001b00 con 0x7fb67c0b1730 2024-03-25T18:36:24.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.940+0000 7fb66affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:24.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.940+0000 7fb66affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:24.941 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.940+0000 7fb66affd640 10 monclient(hunting): dump: 2024-03-25T18:36:24.941 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:24.941 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:24.941 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:24.941 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:24.941 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:24.942 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:24.942 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:24.942 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:24.942 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:24.942 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:24.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 906905204 0 0) 0x7fb67c1421b0 con 0x7fb67c0b4d60 2024-03-25T18:36:24.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 >> v1:172.21.15.190:6789/0 conn(0x7fb67c14be90 legacy=0x7fb67c1341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:24.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 >> v1:172.21.15.99:6789/0 conn(0x7fb67c0b1730 legacy=0x7fb67c1425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:24.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb66affd640 10 monclient: _finish_hunting 0 2024-03-25T18:36:24.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb66affd640 1 monclient: found mon.c 2024-03-25T18:36:24.944 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb66affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:24.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb66affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:36:24.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c140f00 con 0x7fb67c0b4d60 2024-03-25T18:36:24.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb66affd640 10 monclient: _finish_auth 0 2024-03-25T18:36:24.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb66affd640 10 monclient: _check_auth_tickets 2024-03-25T18:36:24.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb66affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:24.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb68091c640 5 monclient: authenticate success, global_id 4724 2024-03-25T18:36:24.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb68091c640 10 monclient: _renew_subs 2024-03-25T18:36:24.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb68091c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:36:24.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb68091c640 1 -- 172.21.15.99:0/1536073591 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67c1406f0 con 0x7fb67c0b4d60 2024-03-25T18:36:24.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb68091c640 10 client.4724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:24.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb68091c640 10 monclient: _renew_subs 2024-03-25T18:36:24.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb68091c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:36:24.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb68091c640 1 -- 172.21.15.99:0/1536073591 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c140380 con 0x7fb67c0b4d60 2024-03-25T18:36:24.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.943+0000 7fb68091c640 1 librados: init done 2024-03-25T18:36:24.946 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.944+0000 7fb68091c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:24.946 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.944+0000 7fb68091c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:24.946 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.944+0000 7fb68091c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:36:24.946 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:24.944+0000 7fb68091c640 1 -- 172.21.15.99:0/1536073591 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67c1401f0 con 0x7fb67c0b4d60 2024-03-25T18:36:25.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.232+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6700035f0 con 0x7fb67c0b4d60 2024-03-25T18:36:25.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.232+0000 7fb66affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:25.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.232+0000 7fb678d99640 4 set_mon_vals no callback set 2024-03-25T18:36:25.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.244+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7fb670003900 con 0x7fb67c0b4d60 2024-03-25T18:36:25.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.244+0000 7fb66affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:25.244 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.244+0000 7fb66affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:25.245 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.244+0000 7fb66affd640 10 monclient: dump: 2024-03-25T18:36:25.245 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:25.245 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:25.245 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:25.245 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:25.245 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:25.245 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:25.245 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:25.245 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:25.245 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:25.245 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:25.246 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.245+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7fb6700198d0 con 0x7fb67c0b4d60 2024-03-25T18:36:25.246 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.245+0000 7fb67a59c640 10 monclient: get_auth_request con 0x7fb65c06df20 auth_method 0 2024-03-25T18:36:25.247 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.246+0000 7fb66affd640 10 client.4724.objecter ms_handle_connect 0x7fb65c06df20 2024-03-25T18:36:25.252 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.251+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2833420883 0 0) 0x7fb6700535a0 con 0x7fb67c0b4d60 2024-03-25T18:36:25.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.251+0000 7fb66affd640 10 client.4724.objecter ms_dispatch 0x7fb67c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:25.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.251+0000 7fb66affd640 3 client.4724.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:25.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.251+0000 7fb66affd640 3 client.4724.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:25.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.251+0000 7fb66affd640 20 client.4724.objecter dump_active .. 0 homeless 2024-03-25T18:36:25.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.251+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb67004eee0 con 0x7fb67c0b4d60 2024-03-25T18:36:25.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.252+0000 7fb66affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:25.253 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.252+0000 7fb66affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:25.481 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.480+0000 7fb68091c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:25.481 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.480+0000 7fb68091c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:25.481 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.480+0000 7fb68091c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:36:25.481 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.480+0000 7fb68091c640 1 -- 172.21.15.99:0/1536073591 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb67c0627b0 con 0x7fb67c0b4d60 2024-03-25T18:36:25.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.507+0000 7fb66affd640 1 -- 172.21.15.99:0/1536073591 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 270404165) 0x7fb670050fe0 con 0x7fb67c0b4d60 2024-03-25T18:36:25.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.507+0000 7fb66affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:25.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.507+0000 7fb66affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:25.508 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:36:25.508 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":11,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:35:20.161948+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:20.161948+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:25.511 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.509+0000 7fb68091c640 10 librados: watch_flush enter 2024-03-25T18:36:25.511 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.509+0000 7fb68091c640 10 librados: watch_flush exit 2024-03-25T18:36:25.511 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.509+0000 7fb68091c640 20 client.4724.objecter shutdown clearing up homeless session... 2024-03-25T18:36:25.511 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.509+0000 7fb68091c640 10 client.4724.objecter successfully canceled tick 2024-03-25T18:36:25.511 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.510+0000 7fb68091c640 1 -- 172.21.15.99:0/1536073591 >> v1:172.21.15.190:6800/4220409699 conn(0x7fb65c06df20 legacy=0x7fb65c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:25.511 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.510+0000 7fb68091c640 10 monclient: shutdown 2024-03-25T18:36:25.511 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.510+0000 7fb68091c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:25.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.510+0000 7fb68091c640 1 -- 172.21.15.99:0/1536073591 >> v1:172.21.15.99:6790/0 conn(0x7fb67c0b4d60 legacy=0x7fb67c133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:25.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.510+0000 7fb68091c640 1 -- 172.21.15.99:0/1536073591 shutdown_connections 2024-03-25T18:36:25.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.510+0000 7fb68091c640 1 -- 172.21.15.99:0/1536073591 >> 172.21.15.99:0/1536073591 conn(0x7fb67c0617a0 msgr2=0x7fb67c13c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:25.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.510+0000 7fb68091c640 1 -- 172.21.15.99:0/1536073591 shutdown_connections 2024-03-25T18:36:25.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.510+0000 7fb68091c640 1 -- 172.21.15.99:0/1536073591 wait complete. 2024-03-25T18:36:25.512 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:25.510+0000 7fb68091c640 1 librados: shutdown 2024-03-25T18:36:25.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:30.030 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:11:30.623 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.035 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:11:30.628 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.046 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:11:30.638 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.049 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:11:30.642 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.055 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:11:30.614 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.059 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:11:30.618 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.060 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:11:30.653 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.071 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:11:30.663 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.077 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:11:30.636 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:32.329 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:32.441 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.440+0000 7ffb6e1a8640 10 monclient: get_monmap_and_config 2024-03-25T18:36:32.441 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.440+0000 7ffb6e1a8640 10 monclient: build_initial_monmap 2024-03-25T18:36:32.441 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.440+0000 7ffb6e1a8640 10 monclient: monmap: 2024-03-25T18:36:32.441 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:32.441 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:32.442 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:36:32.441482+0000 2024-03-25T18:36:32.442 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:36:32.441482+0000 2024-03-25T18:36:32.442 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:32.442 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:32.442 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:32.442 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:36:32.442 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:36:32.442 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:32.443 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.441+0000 7ffb6e1a8640 1 Processor -- start 2024-03-25T18:36:32.443 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.441+0000 7ffb6e1a8640 1 -- start start 2024-03-25T18:36:32.443 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.441+0000 7ffb6e1a8640 10 monclient: init 2024-03-25T18:36:32.443 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.442+0000 7ffb6e1a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:32.444 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.442+0000 7ffb6e1a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:32.444 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.442+0000 7ffb6e1a8640 10 monclient(hunting): picked mon.noname-b con 0x7ffb680a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:32.444 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.442+0000 7ffb6e1a8640 10 monclient(hunting): picked mon.noname-c con 0x7ffb6813f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:32.444 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.442+0000 7ffb6e1a8640 10 monclient(hunting): picked mon.noname-a con 0x7ffb681430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:32.444 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.442+0000 7ffb6e1a8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb68142520 con 0x7ffb681430c0 2024-03-25T18:36:32.444 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.442+0000 7ffb6e1a8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb680b28a0 con 0x7ffb6813f710 2024-03-25T18:36:32.444 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.442+0000 7ffb6e1a8640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb680b2a30 con 0x7ffb680a98f0 2024-03-25T18:36:32.444 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.442+0000 7ffb6e1a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:32.444 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.442+0000 7ffb6e1a8640 10 monclient(hunting): authenticate will time out at 8010.610004s 2024-03-25T18:36:32.445 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.445+0000 7ffb66ffd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7ffb6813f710 0x7ffb681429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:59908/0 (socket says 172.21.15.99:59908) 2024-03-25T18:36:32.446 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.445+0000 7ffb66ffd640 1 -- 172.21.15.99:0/3859439521 learned_addr learned my addr 172.21.15.99:0/3859439521 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:32.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.454+0000 7ffb667fc640 1 -- 172.21.15.99:0/3859439521 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802350883 0 0) 0x7ffb680b28a0 con 0x7ffb6813f710 2024-03-25T18:36:32.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.454+0000 7ffb667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:32.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.454+0000 7ffb667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:32.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.454+0000 7ffb667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:32.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.454+0000 7ffb667fc640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:36:32.455 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.454+0000 7ffb667fc640 1 -- 172.21.15.99:0/3859439521 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb50003520 con 0x7ffb6813f710 2024-03-25T18:36:32.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.455+0000 7ffb667fc640 1 -- 172.21.15.99:0/3859439521 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507789604 0 0) 0x7ffb68142520 con 0x7ffb681430c0 2024-03-25T18:36:32.456 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.455+0000 7ffb667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:32.457 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.455+0000 7ffb667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:32.457 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.455+0000 7ffb667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:32.457 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.455+0000 7ffb667fc640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:36:32.457 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.455+0000 7ffb667fc640 1 -- 172.21.15.99:0/3859439521 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb680b28a0 con 0x7ffb681430c0 2024-03-25T18:36:32.466 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 1 -- 172.21.15.99:0/3859439521 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2729766571 0 0) 0x7ffb50003520 con 0x7ffb6813f710 2024-03-25T18:36:32.466 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 1 -- 172.21.15.99:0/3859439521 >> v1:172.21.15.99:6790/0 conn(0x7ffb680a98f0 legacy=0x7ffb680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:32.466 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 1 -- 172.21.15.99:0/3859439521 >> v1:172.21.15.99:6789/0 conn(0x7ffb681430c0 legacy=0x7ffb6814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:32.466 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 10 monclient: _finish_hunting 0 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 1 monclient: found mon.noname-c 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 1 -- 172.21.15.99:0/3859439521 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb680b44c0 con 0x7ffb6813f710 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 10 monclient: _finish_auth 0 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 10 monclient: _check_auth_tickets 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 1 -- 172.21.15.99:0/3859439521 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ffb5c001690 con 0x7ffb6813f710 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb667fc640 10 monclient: dump: 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:32.467 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:32.468 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:32.468 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:32.468 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:32.468 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:32.468 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:32.468 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:32.468 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb6e1a8640 5 monclient: authenticate success, global_id 4729 2024-03-25T18:36:32.468 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.465+0000 7ffb6e1a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:32.477 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.476+0000 7ffb667fc640 1 -- 172.21.15.99:0/3859439521 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb5c0017f0 con 0x7ffb6813f710 2024-03-25T18:36:32.477 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.476+0000 7ffb667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:32.477 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.476+0000 7ffb667fc640 1 -- 172.21.15.99:0/3859439521 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ffb5c001c00 con 0x7ffb6813f710 2024-03-25T18:36:32.477 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.476+0000 7ffb667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:32.477 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.476+0000 7ffb667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:32.477 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.476+0000 7ffb667fc640 10 monclient: dump: 2024-03-25T18:36:32.477 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:32.477 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:32.477 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:32.477 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:32.477 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:32.477 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:32.478 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:32.478 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:32.478 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:32.478 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:32.478 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.476+0000 7ffb6e1a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:32.478 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.476+0000 7ffb6e1a8640 4 set_mon_vals no callback set 2024-03-25T18:36:32.480 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.476+0000 7ffb6e1a8640 10 monclient: shutdown 2024-03-25T18:36:32.480 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.477+0000 7ffb6e1a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:32.480 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.477+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/3859439521 >> v1:172.21.15.190:6789/0 conn(0x7ffb6813f710 legacy=0x7ffb681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:32.480 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.477+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/3859439521 shutdown_connections 2024-03-25T18:36:32.480 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.477+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/3859439521 >> 172.21.15.99:0/3859439521 conn(0x7ffb680b7ea0 msgr2=0x7ffb680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:32.480 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.477+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/3859439521 shutdown_connections 2024-03-25T18:36:32.480 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.477+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/3859439521 wait complete. 2024-03-25T18:36:32.480 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.477+0000 7ffb6e1a8640 10 monclient: build_initial_monmap 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.477+0000 7ffb6e1a8640 10 monclient: monmap: 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.478+0000 7ffb6e1a8640 1 librados: starting msgr at 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.478+0000 7ffb6e1a8640 1 librados: starting objecter 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.478+0000 7ffb6e1a8640 1 Processor -- start 2024-03-25T18:36:32.481 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.478+0000 7ffb6e1a8640 1 -- start start 2024-03-25T18:36:32.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.478+0000 7ffb6e1a8640 1 librados: setting wanted keys 2024-03-25T18:36:32.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.478+0000 7ffb6e1a8640 1 librados: calling monclient init 2024-03-25T18:36:32.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.478+0000 7ffb6e1a8640 10 monclient: init 2024-03-25T18:36:32.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.478+0000 7ffb6e1a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:32.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.478+0000 7ffb6e1a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:36:32.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.478+0000 7ffb6e1a8640 10 monclient(hunting): picked mon.noname-a con 0x7ffb680a98f0 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:32.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.478+0000 7ffb6e1a8640 10 monclient(hunting): picked mon.noname-b con 0x7ffb6813f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:32.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.479+0000 7ffb6e1a8640 10 monclient(hunting): picked mon.noname-c con 0x7ffb681430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:32.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.479+0000 7ffb6e1a8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb681d7490 con 0x7ffb680a98f0 2024-03-25T18:36:32.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.479+0000 7ffb6e1a8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb681d76c0 con 0x7ffb6813f710 2024-03-25T18:36:32.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.479+0000 7ffb6e1a8640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb681d82f0 con 0x7ffb681430c0 2024-03-25T18:36:32.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.479+0000 7ffb6e1a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:32.482 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.479+0000 7ffb6e1a8640 10 monclient(hunting): authenticate will time out at 8010.646979s 2024-03-25T18:36:32.483 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.482+0000 7ffb67fff640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7ffb681430c0 0x7ffb68133fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:53762/0 (socket says 172.21.15.99:53762) 2024-03-25T18:36:32.483 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.482+0000 7ffb67fff640 1 -- 172.21.15.99:0/1053627947 learned_addr learned my addr 172.21.15.99:0/1053627947 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:32.483 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.483+0000 7ffb4ffff640 10 client.?.objecter ms_handle_connect 0x7ffb681430c0 2024-03-25T18:36:32.483 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.483+0000 7ffb4ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:32.487 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.487+0000 7ffb4ffff640 10 client.?.objecter ms_handle_connect 0x7ffb6813f710 2024-03-25T18:36:32.488 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.487+0000 7ffb4ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:32.489 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.488+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719440813 0 0) 0x7ffb681d82f0 con 0x7ffb681430c0 2024-03-25T18:36:32.489 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.488+0000 7ffb4ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:32.490 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.488+0000 7ffb4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:32.490 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.488+0000 7ffb4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:32.490 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.488+0000 7ffb4ffff640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:36:32.490 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.488+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb54004eb0 con 0x7ffb681430c0 2024-03-25T18:36:32.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.495+0000 7ffb4ffff640 10 client.?.objecter ms_handle_connect 0x7ffb680a98f0 2024-03-25T18:36:32.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.495+0000 7ffb4ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:32.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.501+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 <== mon.2 v1:172.21.15.99:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4287133135 0 0) 0x7ffb54004eb0 con 0x7ffb681430c0 2024-03-25T18:36:32.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.501+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 --> v1:172.21.15.99:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb681d82f0 con 0x7ffb681430c0 2024-03-25T18:36:32.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.501+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 <== mon.2 v1:172.21.15.99:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ffb60001c90 con 0x7ffb681430c0 2024-03-25T18:36:32.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.501+0000 7ffb4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:32.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.501+0000 7ffb4ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:32.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.501+0000 7ffb4ffff640 10 monclient(hunting): dump: 2024-03-25T18:36:32.502 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:32.503 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:32.503 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:32.503 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:32.503 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:32.503 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:32.503 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:32.503 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:32.503 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:32.503 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:32.503 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.501+0000 7ffb4ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:32.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.505+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383775860 0 0) 0x7ffb681d76c0 con 0x7ffb6813f710 2024-03-25T18:36:32.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.505+0000 7ffb4ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:32.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.505+0000 7ffb4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:32.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.505+0000 7ffb4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:32.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.505+0000 7ffb4ffff640 10 monclient(hunting): my global_id is 4732 2024-03-25T18:36:32.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.505+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb54004eb0 con 0x7ffb6813f710 2024-03-25T18:36:32.507 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 <== mon.2 v1:172.21.15.99:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3671829674 0 0) 0x7ffb681d82f0 con 0x7ffb681430c0 2024-03-25T18:36:32.507 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 >> v1:172.21.15.190:6789/0 conn(0x7ffb6813f710 legacy=0x7ffb681383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:32.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 >> v1:172.21.15.99:6789/0 conn(0x7ffb680a98f0 legacy=0x7ffb681d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:32.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb4ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:36:32.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb4ffff640 1 monclient: found mon.c 2024-03-25T18:36:32.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:32.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb4ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:36:32.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 --> v1:172.21.15.99:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb68138af0 con 0x7ffb681430c0 2024-03-25T18:36:32.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb4ffff640 10 monclient: _finish_auth 0 2024-03-25T18:36:32.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb4ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:36:32.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:32.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb6e1a8640 5 monclient: authenticate success, global_id 4730 2024-03-25T18:36:32.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb6e1a8640 10 monclient: _renew_subs 2024-03-25T18:36:32.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb6e1a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:36:32.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/1053627947 --> v1:172.21.15.99:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb681376d0 con 0x7ffb681430c0 2024-03-25T18:36:32.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb6e1a8640 10 client.4730.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:32.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb6e1a8640 10 monclient: _renew_subs 2024-03-25T18:36:32.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb6e1a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:36:32.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/1053627947 --> v1:172.21.15.99:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb68134fc0 con 0x7ffb681430c0 2024-03-25T18:36:32.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.506+0000 7ffb6e1a8640 1 librados: init done 2024-03-25T18:36:32.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.507+0000 7ffb6e1a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:32.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.507+0000 7ffb6e1a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:32.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.507+0000 7ffb6e1a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:36:32.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.507+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/1053627947 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb681d7850 con 0x7ffb681430c0 2024-03-25T18:36:32.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.808+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 <== mon.2 v1:172.21.15.99:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb60003990 con 0x7ffb681430c0 2024-03-25T18:36:32.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.808+0000 7ffb4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:32.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.808+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 <== mon.2 v1:172.21.15.99:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7ffb60003da0 con 0x7ffb681430c0 2024-03-25T18:36:32.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.808+0000 7ffb4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:32.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.808+0000 7ffb4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:32.809 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.808+0000 7ffb4ffff640 10 monclient: dump: 2024-03-25T18:36:32.810 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:32.810 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:32.810 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:32.810 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:32.810 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:32.810 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:32.810 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:32.810 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:32.810 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:32.810 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:32.810 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.808+0000 7ffb65ffb640 4 set_mon_vals no callback set 2024-03-25T18:36:32.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.809+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 <== mon.2 v1:172.21.15.99:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7ffb60003fc0 con 0x7ffb681430c0 2024-03-25T18:36:32.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.809+0000 7ffb677fe640 10 monclient: get_auth_request con 0x7ffb5406dd90 auth_method 0 2024-03-25T18:36:32.811 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.810+0000 7ffb4ffff640 10 client.4730.objecter ms_handle_connect 0x7ffb5406dd90 2024-03-25T18:36:32.816 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.815+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 <== mon.2 v1:172.21.15.99:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2833420883 0 0) 0x7ffb60054ac0 con 0x7ffb681430c0 2024-03-25T18:36:32.816 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.815+0000 7ffb4ffff640 10 client.4730.objecter ms_dispatch 0x7ffb68000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:32.816 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.815+0000 7ffb4ffff640 3 client.4730.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:32.816 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.815+0000 7ffb4ffff640 3 client.4730.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:32.816 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.815+0000 7ffb4ffff640 20 client.4730.objecter dump_active .. 0 homeless 2024-03-25T18:36:32.816 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.815+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 <== mon.2 v1:172.21.15.99:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb600505d0 con 0x7ffb681430c0 2024-03-25T18:36:32.816 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.815+0000 7ffb4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:32.816 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:32.815+0000 7ffb4ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:33.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.075+0000 7ffb6e1a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:33.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.075+0000 7ffb6e1a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:33.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.075+0000 7ffb6e1a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.99:6790/0 2024-03-25T18:36:33.076 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.075+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/1053627947 --> v1:172.21.15.99:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb680627b0 con 0x7ffb681430c0 2024-03-25T18:36:33.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.106+0000 7ffb4ffff640 1 -- 172.21.15.99:0/1053627947 <== mon.2 v1:172.21.15.99:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 270404165) 0x7ffb60052500 con 0x7ffb681430c0 2024-03-25T18:36:33.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.106+0000 7ffb4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:33.107 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.106+0000 7ffb4ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:33.108 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:36:33.108 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":11,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:35:20.161948+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:20.161948+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:33.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.109+0000 7ffb6e1a8640 10 librados: watch_flush enter 2024-03-25T18:36:33.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.109+0000 7ffb6e1a8640 10 librados: watch_flush exit 2024-03-25T18:36:33.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.109+0000 7ffb6e1a8640 20 client.4730.objecter shutdown clearing up homeless session... 2024-03-25T18:36:33.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.109+0000 7ffb6e1a8640 10 client.4730.objecter successfully canceled tick 2024-03-25T18:36:33.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.109+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/1053627947 >> v1:172.21.15.190:6800/4220409699 conn(0x7ffb5406dd90 legacy=0x7ffb54070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:33.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.109+0000 7ffb6e1a8640 10 monclient: shutdown 2024-03-25T18:36:33.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.109+0000 7ffb6e1a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:33.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.109+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/1053627947 >> v1:172.21.15.99:6790/0 conn(0x7ffb681430c0 legacy=0x7ffb68133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:33.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.109+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/1053627947 shutdown_connections 2024-03-25T18:36:33.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.109+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/1053627947 >> 172.21.15.99:0/1053627947 conn(0x7ffb680b7ea0 msgr2=0x7ffb68074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:33.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.109+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/1053627947 shutdown_connections 2024-03-25T18:36:33.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.109+0000 7ffb6e1a8640 1 -- 172.21.15.99:0/1053627947 wait complete. 2024-03-25T18:36:33.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:33.109+0000 7ffb6e1a8640 1 librados: shutdown 2024-03-25T18:36:33.123 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:35.464 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:11:36.056 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:35.513 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:11:36.106 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:35.554 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:11:36.147 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:35.556 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:11:36.149 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:35.557 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:11:36.115 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:35.572 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:11:36.131 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:35.572 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:11:36.165 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:35.594 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:11:36.153 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:35.597 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:11:36.190 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:39.930 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:40.045 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.044+0000 7f62d3e63640 10 monclient: get_monmap_and_config 2024-03-25T18:36:40.045 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.044+0000 7f62d3e63640 10 monclient: build_initial_monmap 2024-03-25T18:36:40.045 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.044+0000 7f62d3e63640 10 monclient: monmap: 2024-03-25T18:36:40.045 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:40.045 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:40.045 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:36:40.045112+0000 2024-03-25T18:36:40.045 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:36:40.045112+0000 2024-03-25T18:36:40.045 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:40.045 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:40.045 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:40.046 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:36:40.046 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:36:40.046 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:40.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.045+0000 7f62d3e63640 1 Processor -- start 2024-03-25T18:36:40.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.045+0000 7f62d3e63640 1 -- start start 2024-03-25T18:36:40.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.045+0000 7f62d3e63640 10 monclient: init 2024-03-25T18:36:40.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.046+0000 7f62d3e63640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:40.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.046+0000 7f62d3e63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:36:40.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.046+0000 7f62d3e63640 10 monclient(hunting): picked mon.noname-b con 0x7f62cc0a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:40.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.046+0000 7f62d3e63640 10 monclient(hunting): picked mon.noname-a con 0x7f62cc13f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:40.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.046+0000 7f62d3e63640 10 monclient(hunting): picked mon.noname-c con 0x7f62cc1430c0 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:40.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.046+0000 7f62d3e63640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc142520 con 0x7f62cc13f710 2024-03-25T18:36:40.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.046+0000 7f62d3e63640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc0b28a0 con 0x7f62cc1430c0 2024-03-25T18:36:40.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.046+0000 7f62d3e63640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc0b2a30 con 0x7f62cc0a98f0 2024-03-25T18:36:40.047 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.046+0000 7f62d3e63640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:40.048 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.046+0000 7f62d3e63640 10 monclient(hunting): authenticate will time out at 8018.214202s 2024-03-25T18:36:40.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.052+0000 7f62d13fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f62cc13f710 0x7f62cc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:52114/0 (socket says 172.21.15.99:52114) 2024-03-25T18:36:40.053 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.052+0000 7f62d13fe640 1 -- 172.21.15.99:0/386254560 learned_addr learned my addr 172.21.15.99:0/386254560 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:40.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.067+0000 7f62d0bfd640 1 -- 172.21.15.99:0/386254560 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799543117 0 0) 0x7f62cc142520 con 0x7f62cc13f710 2024-03-25T18:36:40.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.067+0000 7f62d0bfd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:40.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.067+0000 7f62d0bfd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:40.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.067+0000 7f62d0bfd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:40.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.067+0000 7f62d0bfd640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:36:40.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.067+0000 7f62d0bfd640 1 -- 172.21.15.99:0/386254560 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62b8003520 con 0x7f62cc13f710 2024-03-25T18:36:40.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.068+0000 7f62d0bfd640 1 -- 172.21.15.99:0/386254560 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083215495 0 0) 0x7f62cc0b2a30 con 0x7f62cc0a98f0 2024-03-25T18:36:40.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.068+0000 7f62d0bfd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:40.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.068+0000 7f62d0bfd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:40.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.068+0000 7f62d0bfd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:40.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.068+0000 7f62d0bfd640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:36:40.069 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.068+0000 7f62d0bfd640 1 -- 172.21.15.99:0/386254560 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62cc142520 con 0x7f62cc0a98f0 2024-03-25T18:36:40.071 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.070+0000 7f62d0bfd640 1 -- 172.21.15.99:0/386254560 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684695789 0 0) 0x7f62cc0b28a0 con 0x7f62cc1430c0 2024-03-25T18:36:40.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.070+0000 7f62d0bfd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:40.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.070+0000 7f62d0bfd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:40.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.070+0000 7f62d0bfd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:40.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.070+0000 7f62d0bfd640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:36:40.072 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.070+0000 7f62d0bfd640 1 -- 172.21.15.99:0/386254560 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62cc0b2a30 con 0x7f62cc1430c0 2024-03-25T18:36:40.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 1 -- 172.21.15.99:0/386254560 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1178561732 0 0) 0x7f62b8003520 con 0x7f62cc13f710 2024-03-25T18:36:40.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 1 -- 172.21.15.99:0/386254560 >> v1:172.21.15.99:6790/0 conn(0x7f62cc0a98f0 legacy=0x7f62cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:40.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 1 -- 172.21.15.99:0/386254560 >> v1:172.21.15.190:6789/0 conn(0x7f62cc1430c0 legacy=0x7f62cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:40.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 10 monclient: _finish_hunting 0 2024-03-25T18:36:40.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 1 monclient: found mon.noname-a 2024-03-25T18:36:40.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:40.091 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 1 -- 172.21.15.99:0/386254560 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc0b44c0 con 0x7f62cc13f710 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 10 monclient: _finish_auth 0 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 10 monclient: _check_auth_tickets 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d3e63640 5 monclient: authenticate success, global_id 4725 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d3e63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 1 -- 172.21.15.99:0/386254560 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f62bc001690 con 0x7f62cc13f710 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d0bfd640 10 monclient: dump: 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:40.092 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:40.093 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:40.093 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:40.093 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:40.093 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:40.093 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.090+0000 7f62d3e63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:40.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.108+0000 7f62d0bfd640 1 -- 172.21.15.99:0/386254560 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62bc0017f0 con 0x7f62cc13f710 2024-03-25T18:36:40.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.108+0000 7f62d0bfd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:40.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.108+0000 7f62d0bfd640 1 -- 172.21.15.99:0/386254560 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f62bc001c00 con 0x7f62cc13f710 2024-03-25T18:36:40.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.108+0000 7f62d0bfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:40.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.108+0000 7f62d0bfd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:40.109 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.108+0000 7f62d0bfd640 10 monclient: dump: 2024-03-25T18:36:40.109 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:40.109 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:40.109 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:40.109 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:40.109 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:40.109 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:40.109 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:40.110 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:40.110 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:40.110 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:40.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.108+0000 7f62d3e63640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:40.110 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.108+0000 7f62d3e63640 4 set_mon_vals no callback set 2024-03-25T18:36:40.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.109+0000 7f62d3e63640 10 monclient: shutdown 2024-03-25T18:36:40.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.109+0000 7f62d3e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:40.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.109+0000 7f62d3e63640 1 -- 172.21.15.99:0/386254560 >> v1:172.21.15.99:6789/0 conn(0x7f62cc13f710 legacy=0x7f62cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:40.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.109+0000 7f62d3e63640 1 -- 172.21.15.99:0/386254560 shutdown_connections 2024-03-25T18:36:40.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.109+0000 7f62d3e63640 1 -- 172.21.15.99:0/386254560 >> 172.21.15.99:0/386254560 conn(0x7f62cc0b7ea0 msgr2=0x7f62cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:40.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.109+0000 7f62d3e63640 1 -- 172.21.15.99:0/386254560 shutdown_connections 2024-03-25T18:36:40.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.109+0000 7f62d3e63640 1 -- 172.21.15.99:0/386254560 wait complete. 2024-03-25T18:36:40.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.110+0000 7f62d3e63640 10 monclient: build_initial_monmap 2024-03-25T18:36:40.111 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.110+0000 7f62d3e63640 10 monclient: monmap: 2024-03-25T18:36:40.111 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:40.111 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:40.112 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:36:40.112 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:36:40.112 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:40.112 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:36:40.112 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:40.112 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:36:40.112 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:36:40.112 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:40.113 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.111+0000 7f62d3e63640 1 librados: starting msgr at 2024-03-25T18:36:40.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.111+0000 7f62d3e63640 1 librados: starting objecter 2024-03-25T18:36:40.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.111+0000 7f62d3e63640 1 Processor -- start 2024-03-25T18:36:40.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.111+0000 7f62d3e63640 1 -- start start 2024-03-25T18:36:40.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.111+0000 7f62d3e63640 1 librados: setting wanted keys 2024-03-25T18:36:40.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.111+0000 7f62d3e63640 1 librados: calling monclient init 2024-03-25T18:36:40.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.111+0000 7f62d3e63640 10 monclient: init 2024-03-25T18:36:40.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.111+0000 7f62d3e63640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:40.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.111+0000 7f62d3e63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:40.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.111+0000 7f62d3e63640 10 monclient(hunting): picked mon.noname-c con 0x7f62cc0a98f0 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:40.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.112+0000 7f62d3e63640 10 monclient(hunting): picked mon.noname-b con 0x7f62cc13f710 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:40.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.112+0000 7f62d3e63640 10 monclient(hunting): picked mon.noname-a con 0x7f62cc1430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:40.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.112+0000 7f62d3e63640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc0abd70 con 0x7f62cc1430c0 2024-03-25T18:36:40.114 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.112+0000 7f62d3e63640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc1d7710 con 0x7f62cc13f710 2024-03-25T18:36:40.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.112+0000 7f62d3e63640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc1d8340 con 0x7f62cc0a98f0 2024-03-25T18:36:40.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.112+0000 7f62d3e63640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:40.115 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.112+0000 7f62d3e63640 10 monclient(hunting): authenticate will time out at 8018.280022s 2024-03-25T18:36:40.121 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.120+0000 7f62d13fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f62cc13f710 0x7f62cc1d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:47044/0 (socket says 172.21.15.99:47044) 2024-03-25T18:36:40.121 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.120+0000 7f62d13fe640 1 -- 172.21.15.99:0/934652760 learned_addr learned my addr 172.21.15.99:0/934652760 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:40.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.121+0000 7f62c27fc640 10 client.?.objecter ms_handle_connect 0x7f62cc13f710 2024-03-25T18:36:40.122 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.121+0000 7f62c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:40.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.133+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339646010 0 0) 0x7f62cc1d7710 con 0x7f62cc13f710 2024-03-25T18:36:40.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.133+0000 7f62c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:40.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.133+0000 7f62c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:40.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.133+0000 7f62c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:40.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.133+0000 7f62c27fc640 10 monclient(hunting): my global_id is 4738 2024-03-25T18:36:40.134 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.133+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62b0004eb0 con 0x7f62cc13f710 2024-03-25T18:36:40.137 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.136+0000 7f62c27fc640 10 client.?.objecter ms_handle_connect 0x7f62cc1430c0 2024-03-25T18:36:40.137 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.136+0000 7f62c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:40.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.142+0000 7f62c27fc640 10 client.?.objecter ms_handle_connect 0x7f62cc0a98f0 2024-03-25T18:36:40.143 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.142+0000 7f62c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:40.151 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.151+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567705194 0 0) 0x7f62cc1d8340 con 0x7f62cc0a98f0 2024-03-25T18:36:40.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.151+0000 7f62c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:40.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.151+0000 7f62c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:40.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.151+0000 7f62c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:40.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.151+0000 7f62c27fc640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:36:40.152 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.151+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62cc1d7710 con 0x7f62cc0a98f0 2024-03-25T18:36:40.153 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.152+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2806234179 0 0) 0x7f62b0004eb0 con 0x7f62cc13f710 2024-03-25T18:36:40.153 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.152+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 --> v1:172.21.15.190:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62cc1d8340 con 0x7f62cc13f710 2024-03-25T18:36:40.153 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.152+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f62bc001bc0 con 0x7f62cc13f710 2024-03-25T18:36:40.153 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.152+0000 7f62c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:40.153 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.152+0000 7f62c27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:40.153 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.152+0000 7f62c27fc640 10 monclient(hunting): dump: 2024-03-25T18:36:40.153 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:40.153 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:40.153 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:40.154 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:40.154 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:40.154 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:40.154 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:40.154 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:40.154 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:40.154 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:40.154 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.152+0000 7f62c27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:40.157 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.156+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110602833 0 0) 0x7f62cc0abd70 con 0x7f62cc1430c0 2024-03-25T18:36:40.157 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.156+0000 7f62c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:40.157 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.156+0000 7f62c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:40.157 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.156+0000 7f62c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:40.157 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.156+0000 7f62c27fc640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:36:40.157 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.156+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62b0004eb0 con 0x7f62cc1430c0 2024-03-25T18:36:40.169 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 <== mon.1 v1:172.21.15.190:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3916179677 0 0) 0x7f62cc1d8340 con 0x7f62cc13f710 2024-03-25T18:36:40.169 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 >> v1:172.21.15.99:6790/0 conn(0x7f62cc0a98f0 legacy=0x7f62cc137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:40.169 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 >> v1:172.21.15.99:6789/0 conn(0x7f62cc1430c0 legacy=0x7f62cc1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:40.169 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62c27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:36:40.169 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62c27fc640 1 monclient: found mon.b 2024-03-25T18:36:40.169 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:40.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62c27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:40.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc1d84d0 con 0x7f62cc13f710 2024-03-25T18:36:40.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62c27fc640 10 monclient: _finish_auth 0 2024-03-25T18:36:40.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62c27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:36:40.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:40.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62d3e63640 5 monclient: authenticate success, global_id 4738 2024-03-25T18:36:40.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62d3e63640 10 monclient: _renew_subs 2024-03-25T18:36:40.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62d3e63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:40.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62d3e63640 1 -- 172.21.15.99:0/934652760 --> v1:172.21.15.190:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62cc1d41e0 con 0x7f62cc13f710 2024-03-25T18:36:40.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62d3e63640 10 client.4738.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:40.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62d3e63640 10 monclient: _renew_subs 2024-03-25T18:36:40.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62d3e63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:40.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62d3e63640 1 -- 172.21.15.99:0/934652760 --> v1:172.21.15.190:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62cc1d5150 con 0x7f62cc13f710 2024-03-25T18:36:40.170 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.168+0000 7f62d3e63640 1 librados: init done 2024-03-25T18:36:40.171 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.169+0000 7f62d3e63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:40.171 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.169+0000 7f62d3e63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:40.171 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.169+0000 7f62d3e63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:40.171 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.169+0000 7f62d3e63640 1 -- 172.21.15.99:0/934652760 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62cc0627b0 con 0x7f62cc13f710 2024-03-25T18:36:40.452 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.451+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 <== mon.1 v1:172.21.15.190:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62bc0011f0 con 0x7f62cc13f710 2024-03-25T18:36:40.452 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.451+0000 7f62c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:40.452 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.451+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 <== mon.1 v1:172.21.15.190:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f62bc004cb0 con 0x7f62cc13f710 2024-03-25T18:36:40.452 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.451+0000 7f62c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:40.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.451+0000 7f62c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:40.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.451+0000 7f62c27fc640 10 monclient: dump: 2024-03-25T18:36:40.453 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:40.453 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:40.453 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:40.453 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:40.453 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:40.453 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:40.453 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:40.453 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:40.453 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:40.453 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:40.453 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.451+0000 7f62c3fff640 4 set_mon_vals no callback set 2024-03-25T18:36:40.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.452+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 <== mon.1 v1:172.21.15.190:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f62bc01acc0 con 0x7f62cc13f710 2024-03-25T18:36:40.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.453+0000 7f62d1bff640 10 monclient: get_auth_request con 0x7f62b006dd40 auth_method 0 2024-03-25T18:36:40.454 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.453+0000 7f62c27fc640 10 client.4738.objecter ms_handle_connect 0x7f62b006dd40 2024-03-25T18:36:40.459 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.458+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 <== mon.1 v1:172.21.15.190:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2833420883 0 0) 0x7f62bc0549c0 con 0x7f62cc13f710 2024-03-25T18:36:40.459 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.458+0000 7f62c27fc640 10 client.4738.objecter ms_dispatch 0x7f62cc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:40.459 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.458+0000 7f62c27fc640 3 client.4738.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:40.459 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.458+0000 7f62c27fc640 3 client.4738.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:40.459 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.458+0000 7f62c27fc640 20 client.4738.objecter dump_active .. 0 homeless 2024-03-25T18:36:40.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.460+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 <== mon.1 v1:172.21.15.190:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62bc050300 con 0x7f62cc13f710 2024-03-25T18:36:40.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.460+0000 7f62c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:40.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.460+0000 7f62c27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:40.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.705+0000 7f62d3e63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:40.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.705+0000 7f62d3e63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:40.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.705+0000 7f62d3e63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.190:6789/0 2024-03-25T18:36:40.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.705+0000 7f62d3e63640 1 -- 172.21.15.99:0/934652760 --> v1:172.21.15.190:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62cc1d78a0 con 0x7f62cc13f710 2024-03-25T18:36:40.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.743+0000 7f62c27fc640 1 -- 172.21.15.99:0/934652760 <== mon.1 v1:172.21.15.190:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 270404165) 0x7f62bc052400 con 0x7f62cc13f710 2024-03-25T18:36:40.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.743+0000 7f62c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:40.745 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.743+0000 7f62c27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:40.746 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:36:40.746 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":11,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:35:20.161948+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:20.161948+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:40.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.746+0000 7f62d3e63640 10 librados: watch_flush enter 2024-03-25T18:36:40.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.746+0000 7f62d3e63640 10 librados: watch_flush exit 2024-03-25T18:36:40.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.746+0000 7f62d3e63640 20 client.4738.objecter shutdown clearing up homeless session... 2024-03-25T18:36:40.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.746+0000 7f62d3e63640 10 client.4738.objecter successfully canceled tick 2024-03-25T18:36:40.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.746+0000 7f62d3e63640 1 -- 172.21.15.99:0/934652760 >> v1:172.21.15.190:6800/4220409699 conn(0x7f62b006dd40 legacy=0x7f62b0070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:40.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.746+0000 7f62d3e63640 10 monclient: shutdown 2024-03-25T18:36:40.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.746+0000 7f62d3e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:40.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.746+0000 7f62d3e63640 1 -- 172.21.15.99:0/934652760 >> v1:172.21.15.190:6789/0 conn(0x7f62cc13f710 legacy=0x7f62cc1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:40.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.746+0000 7f62d3e63640 1 -- 172.21.15.99:0/934652760 shutdown_connections 2024-03-25T18:36:40.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.746+0000 7f62d3e63640 1 -- 172.21.15.99:0/934652760 >> 172.21.15.99:0/934652760 conn(0x7f62cc0b7ea0 msgr2=0x7f62cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:40.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.746+0000 7f62d3e63640 1 -- 172.21.15.99:0/934652760 shutdown_connections 2024-03-25T18:36:40.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.746+0000 7f62d3e63640 1 -- 172.21.15.99:0/934652760 wait complete. 2024-03-25T18:36:40.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:40.746+0000 7f62d3e63640 1 librados: shutdown 2024-03-25T18:36:40.759 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:47.566 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:47.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.679+0000 7f6a2b858640 10 monclient: get_monmap_and_config 2024-03-25T18:36:47.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.679+0000 7f6a2b858640 10 monclient: build_initial_monmap 2024-03-25T18:36:47.681 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.679+0000 7f6a2b858640 10 monclient: monmap: 2024-03-25T18:36:47.681 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:47.681 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:47.681 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:36:47.680832+0000 2024-03-25T18:36:47.681 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:36:47.680832+0000 2024-03-25T18:36:47.681 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:47.681 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:47.681 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:47.682 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:36:47.682 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:36:47.682 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:47.682 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.680+0000 7f6a2b858640 1 Processor -- start 2024-03-25T18:36:47.683 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.680+0000 7f6a2b858640 1 -- start start 2024-03-25T18:36:47.683 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.680+0000 7f6a2b858640 10 monclient: init 2024-03-25T18:36:47.683 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.681+0000 7f6a2b858640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:47.683 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.681+0000 7f6a2b858640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:36:47.683 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.681+0000 7f6a2b858640 10 monclient(hunting): picked mon.noname-c con 0x7f6a240a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:47.683 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.681+0000 7f6a2b858640 10 monclient(hunting): picked mon.noname-a con 0x7f6a2413f710 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:47.683 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.681+0000 7f6a2b858640 10 monclient(hunting): picked mon.noname-b con 0x7f6a241430c0 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:47.683 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.681+0000 7f6a2b858640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a24142520 con 0x7f6a2413f710 2024-03-25T18:36:47.683 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.681+0000 7f6a2b858640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a240b28a0 con 0x7f6a240a98f0 2024-03-25T18:36:47.683 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.681+0000 7f6a2b858640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a240b2a30 con 0x7f6a241430c0 2024-03-25T18:36:47.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.681+0000 7f6a2b858640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:47.684 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.681+0000 7f6a2b858640 10 monclient(hunting): authenticate will time out at 8025.849332s 2024-03-25T18:36:47.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.691+0000 7f6a28df3640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f6a2413f710 0x7f6a241429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51250/0 (socket says 172.21.15.99:51250) 2024-03-25T18:36:47.691 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.691+0000 7f6a28df3640 1 -- 172.21.15.99:0/1740298967 learned_addr learned my addr 172.21.15.99:0/1740298967 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:47.706 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.705+0000 7f6a1bfff640 1 -- 172.21.15.99:0/1740298967 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549092911 0 0) 0x7f6a24142520 con 0x7f6a2413f710 2024-03-25T18:36:47.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.706+0000 7f6a1bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:47.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.706+0000 7f6a1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:47.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.706+0000 7f6a1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:47.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.706+0000 7f6a1bfff640 10 monclient(hunting): my global_id is 4731 2024-03-25T18:36:47.707 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.706+0000 7f6a1bfff640 1 -- 172.21.15.99:0/1740298967 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a10003520 con 0x7f6a2413f710 2024-03-25T18:36:47.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.714+0000 7f6a1bfff640 1 -- 172.21.15.99:0/1740298967 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186925959 0 0) 0x7f6a240b28a0 con 0x7f6a240a98f0 2024-03-25T18:36:47.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.714+0000 7f6a1bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:47.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.714+0000 7f6a1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:47.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.714+0000 7f6a1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:47.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.714+0000 7f6a1bfff640 10 monclient(hunting): my global_id is 4741 2024-03-25T18:36:47.715 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.714+0000 7f6a1bfff640 1 -- 172.21.15.99:0/1740298967 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a24142520 con 0x7f6a240a98f0 2024-03-25T18:36:47.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.724+0000 7f6a1bfff640 1 -- 172.21.15.99:0/1740298967 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750038643 0 0) 0x7f6a240b2a30 con 0x7f6a241430c0 2024-03-25T18:36:47.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.724+0000 7f6a1bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:36:47.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.724+0000 7f6a1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:47.725 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.724+0000 7f6a1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:47.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.724+0000 7f6a1bfff640 10 monclient(hunting): my global_id is 4739 2024-03-25T18:36:47.726 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.724+0000 7f6a1bfff640 1 -- 172.21.15.99:0/1740298967 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a240b28a0 con 0x7f6a241430c0 2024-03-25T18:36:47.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 1 -- 172.21.15.99:0/1740298967 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1657541786 0 0) 0x7f6a10003520 con 0x7f6a2413f710 2024-03-25T18:36:47.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 1 -- 172.21.15.99:0/1740298967 >> v1:172.21.15.99:6790/0 conn(0x7f6a241430c0 legacy=0x7f6a2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:47.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 1 -- 172.21.15.99:0/1740298967 >> v1:172.21.15.190:6789/0 conn(0x7f6a240a98f0 legacy=0x7f6a240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:47.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:36:47.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 1 monclient: found mon.noname-a 2024-03-25T18:36:47.747 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.99:6789/0 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 1 -- 172.21.15.99:0/1740298967 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a240b44c0 con 0x7f6a2413f710 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 10 monclient: _finish_auth 0 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 1 -- 172.21.15.99:0/1740298967 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6a14001690 con 0x7f6a2413f710 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a2b858640 5 monclient: authenticate success, global_id 4731 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a2b858640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a1bfff640 10 monclient: dump: 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:47.748 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:47.749 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:47.749 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:47.749 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:47.749 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:47.749 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:47.749 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:47.749 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.746+0000 7f6a2b858640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:47.781 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.780+0000 7f6a1bfff640 1 -- 172.21.15.99:0/1740298967 <== mon.0 v1:172.21.15.99:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a140017f0 con 0x7f6a2413f710 2024-03-25T18:36:47.781 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.780+0000 7f6a1bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:47.781 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.780+0000 7f6a1bfff640 1 -- 172.21.15.99:0/1740298967 <== mon.0 v1:172.21.15.99:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6a14001c00 con 0x7f6a2413f710 2024-03-25T18:36:47.781 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.780+0000 7f6a1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:47.781 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.780+0000 7f6a1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:47.781 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.780+0000 7f6a1bfff640 10 monclient: dump: 2024-03-25T18:36:47.782 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:47.782 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:47.782 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:47.782 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:47.782 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:47.782 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:47.782 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:47.782 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:47.782 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:47.782 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:47.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.780+0000 7f6a2b858640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:47.782 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.780+0000 7f6a2b858640 4 set_mon_vals no callback set 2024-03-25T18:36:47.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.781+0000 7f6a2b858640 10 monclient: shutdown 2024-03-25T18:36:47.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.781+0000 7f6a2b858640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:47.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.781+0000 7f6a2b858640 1 -- 172.21.15.99:0/1740298967 >> v1:172.21.15.99:6789/0 conn(0x7f6a2413f710 legacy=0x7f6a241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:47.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.781+0000 7f6a2b858640 1 -- 172.21.15.99:0/1740298967 shutdown_connections 2024-03-25T18:36:47.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.781+0000 7f6a2b858640 1 -- 172.21.15.99:0/1740298967 >> 172.21.15.99:0/1740298967 conn(0x7f6a240b7ea0 msgr2=0x7f6a240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:47.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.781+0000 7f6a2b858640 1 -- 172.21.15.99:0/1740298967 shutdown_connections 2024-03-25T18:36:47.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.781+0000 7f6a2b858640 1 -- 172.21.15.99:0/1740298967 wait complete. 2024-03-25T18:36:47.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.782+0000 7f6a2b858640 10 monclient: build_initial_monmap 2024-03-25T18:36:47.784 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.782+0000 7f6a2b858640 10 monclient: monmap: 2024-03-25T18:36:47.784 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:47.784 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:47.784 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:36:47.784 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:36:47.785 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:47.785 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:36:47.785 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:47.785 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:36:47.785 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:36:47.785 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:47.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.783+0000 7f6a2b858640 1 librados: starting msgr at 2024-03-25T18:36:47.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.783+0000 7f6a2b858640 1 librados: starting objecter 2024-03-25T18:36:47.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.783+0000 7f6a2b858640 1 Processor -- start 2024-03-25T18:36:47.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.783+0000 7f6a2b858640 1 -- start start 2024-03-25T18:36:47.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.783+0000 7f6a2b858640 1 librados: setting wanted keys 2024-03-25T18:36:47.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.783+0000 7f6a2b858640 1 librados: calling monclient init 2024-03-25T18:36:47.785 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.783+0000 7f6a2b858640 10 monclient: init 2024-03-25T18:36:47.787 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.784+0000 7f6a2b858640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:47.787 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.784+0000 7f6a2b858640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:47.787 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.784+0000 7f6a2b858640 10 monclient(hunting): picked mon.noname-b con 0x7f6a240a98f0 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:47.787 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.784+0000 7f6a2b858640 10 monclient(hunting): picked mon.noname-c con 0x7f6a2413f710 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:47.787 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.784+0000 7f6a2b858640 10 monclient(hunting): picked mon.noname-a con 0x7f6a241430c0 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:47.787 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.784+0000 7f6a2b858640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a240abac0 con 0x7f6a241430c0 2024-03-25T18:36:47.787 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.784+0000 7f6a2b858640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a240abda0 con 0x7f6a240a98f0 2024-03-25T18:36:47.787 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.784+0000 7f6a2b858640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a241380e0 con 0x7f6a2413f710 2024-03-25T18:36:47.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.784+0000 7f6a2b858640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:47.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.784+0000 7f6a2b858640 10 monclient(hunting): authenticate will time out at 8025.952216s 2024-03-25T18:36:47.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.784+0000 7f6a28df3640 1 --1- >> v1:172.21.15.99:6790/0 conn(0x7f6a2413f710 0x7f6a24138b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.99:6790/0 says I am v1:172.21.15.99:34468/0 (socket says 172.21.15.99:34468) 2024-03-25T18:36:47.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.784+0000 7f6a28df3640 1 -- 172.21.15.99:0/2977445810 learned_addr learned my addr 172.21.15.99:0/2977445810 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:47.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.785+0000 7f6a19ffb640 10 client.?.objecter ms_handle_connect 0x7f6a2413f710 2024-03-25T18:36:47.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.785+0000 7f6a19ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:47.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.786+0000 7f6a19ffb640 10 client.?.objecter ms_handle_connect 0x7f6a240a98f0 2024-03-25T18:36:47.788 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.786+0000 7f6a19ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:47.877 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.876+0000 7f6a19ffb640 10 client.?.objecter ms_handle_connect 0x7f6a241430c0 2024-03-25T18:36:47.877 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.876+0000 7f6a19ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:47.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.889+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258043675 0 0) 0x7f6a240abac0 con 0x7f6a241430c0 2024-03-25T18:36:47.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.889+0000 7f6a19ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:36:47.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.889+0000 7f6a19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:47.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.889+0000 7f6a19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:47.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.889+0000 7f6a19ffb640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:36:47.890 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.889+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a08004eb0 con 0x7f6a241430c0 2024-03-25T18:36:47.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.899+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 397857838 0 0) 0x7f6a08004eb0 con 0x7f6a241430c0 2024-03-25T18:36:47.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.899+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a240abac0 con 0x7f6a241430c0 2024-03-25T18:36:47.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.899+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6a1c001bd0 con 0x7f6a241430c0 2024-03-25T18:36:47.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.899+0000 7f6a19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:47.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.899+0000 7f6a19ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:47.900 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.899+0000 7f6a19ffb640 10 monclient(hunting): dump: 2024-03-25T18:36:47.901 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:47.901 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:47.901 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:47.901 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:47.901 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:47.901 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:47.901 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:47.901 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:47.901 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:47.901 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:47.901 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.899+0000 7f6a19ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:47.914 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.913+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901932554 0 0) 0x7f6a240abda0 con 0x7f6a240a98f0 2024-03-25T18:36:47.914 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.913+0000 7f6a19ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:36:47.914 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.913+0000 7f6a19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:47.914 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.913+0000 7f6a19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:47.914 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.913+0000 7f6a19ffb640 10 monclient(hunting): my global_id is 4744 2024-03-25T18:36:47.914 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.913+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a08004eb0 con 0x7f6a240a98f0 2024-03-25T18:36:47.921 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 15013265 0 0) 0x7f6a240abac0 con 0x7f6a241430c0 2024-03-25T18:36:47.921 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 >> v1:172.21.15.99:6790/0 conn(0x7f6a2413f710 legacy=0x7f6a24138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:47.921 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 >> v1:172.21.15.190:6789/0 conn(0x7f6a240a98f0 legacy=0x7f6a240ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:47.921 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a19ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:36:47.921 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a19ffb640 1 monclient: found mon.a 2024-03-25T18:36:47.921 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:47.921 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a19ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:47.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a24137d90 con 0x7f6a241430c0 2024-03-25T18:36:47.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a19ffb640 10 monclient: _finish_auth 0 2024-03-25T18:36:47.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a19ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:36:47.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:47.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a2b858640 5 monclient: authenticate success, global_id 4734 2024-03-25T18:36:47.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a2b858640 10 monclient: _renew_subs 2024-03-25T18:36:47.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a2b858640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:47.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a2b858640 1 -- 172.21.15.99:0/2977445810 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a24139240 con 0x7f6a241430c0 2024-03-25T18:36:47.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a2b858640 10 client.4734.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:47.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a2b858640 10 monclient: _renew_subs 2024-03-25T18:36:47.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a2b858640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:47.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a2b858640 1 -- 172.21.15.99:0/2977445810 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a24136830 con 0x7f6a241430c0 2024-03-25T18:36:47.922 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.920+0000 7f6a2b858640 1 librados: init done 2024-03-25T18:36:47.923 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.921+0000 7f6a2b858640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:47.923 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.921+0000 7f6a2b858640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:47.923 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.921+0000 7f6a2b858640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:47.923 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:47.921+0000 7f6a2b858640 1 -- 172.21.15.99:0/2977445810 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a240627b0 con 0x7f6a241430c0 2024-03-25T18:36:48.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.231+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a1c0020a0 con 0x7f6a241430c0 2024-03-25T18:36:48.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.231+0000 7f6a19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:48.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.231+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f6a1c003b30 con 0x7f6a241430c0 2024-03-25T18:36:48.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.231+0000 7f6a19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:48.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.231+0000 7f6a19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:48.232 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.231+0000 7f6a19ffb640 10 monclient: dump: 2024-03-25T18:36:48.232 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:48.232 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:48.232 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:48.233 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:48.233 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:48.233 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:48.233 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:48.233 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:48.233 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:48.233 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:48.233 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.231+0000 7f6a1b7fe640 4 set_mon_vals no callback set 2024-03-25T18:36:48.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.232+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f6a1c003d50 con 0x7f6a241430c0 2024-03-25T18:36:48.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.232+0000 7f6a295f4640 10 monclient: get_auth_request con 0x7f6a0806dcb0 auth_method 0 2024-03-25T18:36:48.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.233+0000 7f6a19ffb640 10 client.4734.objecter ms_handle_connect 0x7f6a0806dcb0 2024-03-25T18:36:48.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.238+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2833420883 0 0) 0x7f6a1c053930 con 0x7f6a241430c0 2024-03-25T18:36:48.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.238+0000 7f6a19ffb640 10 client.4734.objecter ms_dispatch 0x7f6a24000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:48.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.238+0000 7f6a19ffb640 3 client.4734.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:48.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.238+0000 7f6a19ffb640 3 client.4734.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:48.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.238+0000 7f6a19ffb640 20 client.4734.objecter dump_active .. 0 homeless 2024-03-25T18:36:48.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.238+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a1c04f270 con 0x7f6a241430c0 2024-03-25T18:36:48.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.238+0000 7f6a19ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:48.239 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.238+0000 7f6a19ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:48.495 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.494+0000 7f6a2b858640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:48.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.494+0000 7f6a2b858640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:48.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.494+0000 7f6a2b858640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:48.496 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.494+0000 7f6a2b858640 1 -- 172.21.15.99:0/2977445810 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a2413f070 con 0x7f6a241430c0 2024-03-25T18:36:48.522 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.521+0000 7f6a19ffb640 1 -- 172.21.15.99:0/2977445810 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 270404165) 0x7f6a1c051370 con 0x7f6a241430c0 2024-03-25T18:36:48.522 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.521+0000 7f6a19ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:48.523 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.521+0000 7f6a19ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:48.523 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:36:48.523 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":11,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:35:20.161948+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:20.161948+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:48.525 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.524+0000 7f6a2b858640 10 librados: watch_flush enter 2024-03-25T18:36:48.525 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.524+0000 7f6a2b858640 10 librados: watch_flush exit 2024-03-25T18:36:48.525 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.524+0000 7f6a2b858640 20 client.4734.objecter shutdown clearing up homeless session... 2024-03-25T18:36:48.525 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.524+0000 7f6a2b858640 10 client.4734.objecter successfully canceled tick 2024-03-25T18:36:48.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.524+0000 7f6a2b858640 1 -- 172.21.15.99:0/2977445810 >> v1:172.21.15.190:6800/4220409699 conn(0x7f6a0806dcb0 legacy=0x7f6a08070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:48.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.524+0000 7f6a2b858640 10 monclient: shutdown 2024-03-25T18:36:48.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.524+0000 7f6a2b858640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:48.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.524+0000 7f6a2b858640 1 -- 172.21.15.99:0/2977445810 >> v1:172.21.15.99:6789/0 conn(0x7f6a241430c0 legacy=0x7f6a241e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:48.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.524+0000 7f6a2b858640 1 -- 172.21.15.99:0/2977445810 shutdown_connections 2024-03-25T18:36:48.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.524+0000 7f6a2b858640 1 -- 172.21.15.99:0/2977445810 >> 172.21.15.99:0/2977445810 conn(0x7f6a240b7ea0 msgr2=0x7f6a24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:48.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.525+0000 7f6a2b858640 1 -- 172.21.15.99:0/2977445810 shutdown_connections 2024-03-25T18:36:48.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.525+0000 7f6a2b858640 1 -- 172.21.15.99:0/2977445810 wait complete. 2024-03-25T18:36:48.526 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:48.525+0000 7f6a2b858640 1 librados: shutdown 2024-03-25T18:36:48.536 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:55.343 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:55.458 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.457+0000 7f990ae2a640 10 monclient: get_monmap_and_config 2024-03-25T18:36:55.458 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.457+0000 7f990ae2a640 10 monclient: build_initial_monmap 2024-03-25T18:36:55.458 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.457+0000 7f990ae2a640 10 monclient: monmap: 2024-03-25T18:36:55.459 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:55.459 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:55.459 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:36:55.458509+0000 2024-03-25T18:36:55.459 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:36:55.458509+0000 2024-03-25T18:36:55.459 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:55.459 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:55.459 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:55.459 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-c 2024-03-25T18:36:55.460 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-b 2024-03-25T18:36:55.460 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:55.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.458+0000 7f990ae2a640 1 Processor -- start 2024-03-25T18:36:55.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.458+0000 7f990ae2a640 1 -- start start 2024-03-25T18:36:55.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.458+0000 7f990ae2a640 10 monclient: init 2024-03-25T18:36:55.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.458+0000 7f990ae2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:55.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.458+0000 7f990ae2a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:55.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.458+0000 7f990ae2a640 10 monclient(hunting): picked mon.noname-c con 0x7f99040b4d60 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:55.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.458+0000 7f990ae2a640 10 monclient(hunting): picked mon.noname-b con 0x7f99040b1730 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:55.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.458+0000 7f990ae2a640 10 monclient(hunting): picked mon.noname-a con 0x7f990414be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:55.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.458+0000 7f990ae2a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99040af940 con 0x7f990414be90 2024-03-25T18:36:55.461 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.458+0000 7f990ae2a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99040b0a70 con 0x7f99040b4d60 2024-03-25T18:36:55.462 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.458+0000 7f990ae2a640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99040b0d50 con 0x7f99040b1730 2024-03-25T18:36:55.462 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.458+0000 7f990ae2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:55.462 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.458+0000 7f990ae2a640 10 monclient(hunting): authenticate will time out at 8033.626845s 2024-03-25T18:36:55.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.464+0000 7f9908bc6640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f99040b4d60 0x7f990413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:43898/0 (socket says 172.21.15.99:43898) 2024-03-25T18:36:55.465 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.464+0000 7f9908bc6640 1 -- 172.21.15.99:0/204132162 learned_addr learned my addr 172.21.15.99:0/204132162 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:55.476 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.475+0000 7f99037fe640 1 -- 172.21.15.99:0/204132162 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119421871 0 0) 0x7f99040b0a70 con 0x7f99040b4d60 2024-03-25T18:36:55.476 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.475+0000 7f99037fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:55.476 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.475+0000 7f99037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:55.476 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.475+0000 7f99037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:55.476 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.475+0000 7f99037fe640 10 monclient(hunting): my global_id is 4747 2024-03-25T18:36:55.476 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.475+0000 7f99037fe640 1 -- 172.21.15.99:0/204132162 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ec003520 con 0x7f99040b4d60 2024-03-25T18:36:55.485 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 1 -- 172.21.15.99:0/204132162 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3548373688 0 0) 0x7f98ec003520 con 0x7f99040b4d60 2024-03-25T18:36:55.485 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 1 -- 172.21.15.99:0/204132162 >> v1:172.21.15.99:6790/0 conn(0x7f99040b1730 legacy=0x7f99040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:55.485 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 1 -- 172.21.15.99:0/204132162 >> v1:172.21.15.99:6789/0 conn(0x7f990414be90 legacy=0x7f990414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:36:55.485 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 10 monclient: _finish_hunting 0 2024-03-25T18:36:55.485 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 1 monclient: found mon.noname-c 2024-03-25T18:36:55.486 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:55.486 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.190:6789/0 2024-03-25T18:36:55.486 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 1 -- 172.21.15.99:0/204132162 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99040b1030 con 0x7f99040b4d60 2024-03-25T18:36:55.486 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 10 monclient: _finish_auth 0 2024-03-25T18:36:55.486 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 10 monclient: _check_auth_tickets 2024-03-25T18:36:55.486 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:55.487 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 1 -- 172.21.15.99:0/204132162 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f98f4001a10 con 0x7f99040b4d60 2024-03-25T18:36:55.487 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:55.487 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:55.487 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f99037fe640 10 monclient: dump: 2024-03-25T18:36:55.487 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:55.487 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:55.487 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:55.487 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:55.487 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:55.487 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:55.487 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:55.487 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:55.487 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:55.488 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:55.489 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f990ae2a640 5 monclient: authenticate success, global_id 4747 2024-03-25T18:36:55.489 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.483+0000 7f990ae2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:55.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.501+0000 7f99037fe640 1 -- 172.21.15.99:0/204132162 <== mon.1 v1:172.21.15.190:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98f4001be0 con 0x7f99040b4d60 2024-03-25T18:36:55.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.501+0000 7f99037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:55.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.501+0000 7f99037fe640 1 -- 172.21.15.99:0/204132162 <== mon.1 v1:172.21.15.190:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f98f4002030 con 0x7f99040b4d60 2024-03-25T18:36:55.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.501+0000 7f990ae2a640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:55.502 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.501+0000 7f990ae2a640 4 set_mon_vals no callback set 2024-03-25T18:36:55.503 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.501+0000 7f99037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:55.503 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.501+0000 7f99037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:55.503 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.501+0000 7f99037fe640 10 monclient: dump: 2024-03-25T18:36:55.503 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:55.503 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:55.503 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:55.503 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:55.503 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:55.503 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:55.503 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:55.504 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:55.504 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:55.504 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:55.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.502+0000 7f990ae2a640 10 monclient: shutdown 2024-03-25T18:36:55.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.502+0000 7f990ae2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:55.506 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.502+0000 7f990ae2a640 1 -- 172.21.15.99:0/204132162 >> v1:172.21.15.190:6789/0 conn(0x7f99040b4d60 legacy=0x7f990413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:55.507 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.502+0000 7f990ae2a640 1 -- 172.21.15.99:0/204132162 shutdown_connections 2024-03-25T18:36:55.507 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.502+0000 7f990ae2a640 1 -- 172.21.15.99:0/204132162 >> 172.21.15.99:0/204132162 conn(0x7f99040617a0 msgr2=0x7f9904061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:55.507 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.502+0000 7f990ae2a640 1 -- 172.21.15.99:0/204132162 shutdown_connections 2024-03-25T18:36:55.507 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.502+0000 7f990ae2a640 1 -- 172.21.15.99:0/204132162 wait complete. 2024-03-25T18:36:55.507 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.503+0000 7f990ae2a640 10 monclient: build_initial_monmap 2024-03-25T18:36:55.507 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.503+0000 7f990ae2a640 10 monclient: monmap: 2024-03-25T18:36:55.507 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2024-03-25T18:36:55.507 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:55.507 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2024-03-25T18:36:55.507 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2024-03-25T18:36:55.507 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:55.508 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2024-03-25T18:36:55.508 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a 2024-03-25T18:36:55.508 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.noname-b 2024-03-25T18:36:55.508 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.noname-c 2024-03-25T18:36:55.508 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.503+0000 7f990ae2a640 1 librados: starting msgr at 2024-03-25T18:36:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.503+0000 7f990ae2a640 1 librados: starting objecter 2024-03-25T18:36:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.503+0000 7f990ae2a640 1 Processor -- start 2024-03-25T18:36:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.503+0000 7f990ae2a640 1 -- start start 2024-03-25T18:36:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.503+0000 7f990ae2a640 1 librados: setting wanted keys 2024-03-25T18:36:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.503+0000 7f990ae2a640 1 librados: calling monclient init 2024-03-25T18:36:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.503+0000 7f990ae2a640 10 monclient: init 2024-03-25T18:36:55.508 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.504+0000 7f990ae2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:55.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.504+0000 7f990ae2a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:55.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.504+0000 7f990ae2a640 10 monclient(hunting): picked mon.noname-b con 0x7f99040b1730 addr v1:172.21.15.190:6789/0 2024-03-25T18:36:55.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.504+0000 7f990ae2a640 10 monclient(hunting): picked mon.noname-c con 0x7f99040b4d60 addr v1:172.21.15.99:6790/0 2024-03-25T18:36:55.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.504+0000 7f990ae2a640 10 monclient(hunting): picked mon.noname-a con 0x7f990414be90 addr v1:172.21.15.99:6789/0 2024-03-25T18:36:55.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.504+0000 7f990ae2a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9904140760 con 0x7f990414be90 2024-03-25T18:36:55.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.504+0000 7f990ae2a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99041435b0 con 0x7f99040b1730 2024-03-25T18:36:55.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.504+0000 7f990ae2a640 1 -- --> v1:172.21.15.99:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9904140380 con 0x7f99040b4d60 2024-03-25T18:36:55.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.504+0000 7f990ae2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:55.509 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.504+0000 7f990ae2a640 10 monclient(hunting): authenticate will time out at 8033.672726s 2024-03-25T18:36:55.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.506+0000 7f9908bc6640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f99040b1730 0x7f99041d7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.99:43964/0 (socket says 172.21.15.99:43964) 2024-03-25T18:36:55.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.506+0000 7f9908bc6640 1 -- 172.21.15.99:0/2754734893 learned_addr learned my addr 172.21.15.99:0/2754734893 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-25T18:36:55.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.508+0000 7f99017fa640 10 client.?.objecter ms_handle_connect 0x7f99040b1730 2024-03-25T18:36:55.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.508+0000 7f99017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:55.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.508+0000 7f99017fa640 10 client.?.objecter ms_handle_connect 0x7f99040b4d60 2024-03-25T18:36:55.510 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.508+0000 7f99017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:55.548 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:11:56.140 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:55.553 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:11:56.146 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:55.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.559+0000 7f99017fa640 10 client.?.objecter ms_handle_connect 0x7f990414be90 2024-03-25T18:36:55.560 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.559+0000 7f99017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:55.571 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.570+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521707384 0 0) 0x7f9904140760 con 0x7f990414be90 2024-03-25T18:36:55.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.570+0000 7f99017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:55.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.570+0000 7f99017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:55.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.570+0000 7f99017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:55.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.570+0000 7f99017fa640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:36:55.572 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.570+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98f0004eb0 con 0x7f990414be90 2024-03-25T18:36:55.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.589+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3233264256 0 0) 0x7f98f0004eb0 con 0x7f990414be90 2024-03-25T18:36:55.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.589+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 --> v1:172.21.15.99:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9904140760 con 0x7f990414be90 2024-03-25T18:36:55.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.589+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f98fc001750 con 0x7f990414be90 2024-03-25T18:36:55.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.589+0000 7f99017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:55.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.589+0000 7f99017fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:55.590 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.589+0000 7f99017fa640 10 monclient(hunting): dump: 2024-03-25T18:36:55.590 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:55.590 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:55.591 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:55.591 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:55.591 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:55.591 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:55.591 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:55.591 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:55.591 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:55.591 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:55.591 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.589+0000 7f99017fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:55.595 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:11:56.188 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:55.599 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:11:56.191 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:55.602 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:11:56.195 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:55.608 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:11:56.167 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:55.618 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:11:56.177 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:55.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.618+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607618468 0 0) 0x7f99041435b0 con 0x7f99040b1730 2024-03-25T18:36:55.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.618+0000 7f99017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:55.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.618+0000 7f99017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:55.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.618+0000 7f99017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:55.619 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.618+0000 7f99017fa640 10 monclient(hunting): my global_id is 4750 2024-03-25T18:36:55.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.618+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98f0004eb0 con 0x7f99040b1730 2024-03-25T18:36:55.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.619+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 <== mon.2 v1:172.21.15.99:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023646794 0 0) 0x7f9904140380 con 0x7f99040b4d60 2024-03-25T18:36:55.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.619+0000 7f99017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:36:55.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.619+0000 7f99017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:55.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.619+0000 7f99017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:55.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.619+0000 7f99017fa640 10 monclient(hunting): my global_id is 4745 2024-03-25T18:36:55.620 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.619+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 --> v1:172.21.15.99:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99041435b0 con 0x7f99040b4d60 2024-03-25T18:36:55.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 <== mon.0 v1:172.21.15.99:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1782642379 0 0) 0x7f9904140760 con 0x7f990414be90 2024-03-25T18:36:55.623 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 >> v1:172.21.15.99:6790/0 conn(0x7f99040b4d60 legacy=0x7f9904142ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:55.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 >> v1:172.21.15.190:6789/0 conn(0x7f99040b1730 legacy=0x7f99041d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:55.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f99017fa640 10 monclient: _finish_hunting 0 2024-03-25T18:36:55.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f99017fa640 1 monclient: found mon.a 2024-03-25T18:36:55.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f99017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:55.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f99017fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:55.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9904140510 con 0x7f990414be90 2024-03-25T18:36:55.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f99017fa640 10 monclient: _finish_auth 0 2024-03-25T18:36:55.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f99017fa640 10 monclient: _check_auth_tickets 2024-03-25T18:36:55.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f99017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:55.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f990ae2a640 5 monclient: authenticate success, global_id 4737 2024-03-25T18:36:55.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f990ae2a640 10 monclient: _renew_subs 2024-03-25T18:36:55.624 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f990ae2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:55.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f990ae2a640 1 -- 172.21.15.99:0/2754734893 --> v1:172.21.15.99:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99041382b0 con 0x7f990414be90 2024-03-25T18:36:55.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f990ae2a640 10 client.4737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:55.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f990ae2a640 10 monclient: _renew_subs 2024-03-25T18:36:55.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f990ae2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:55.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f990ae2a640 1 -- 172.21.15.99:0/2754734893 --> v1:172.21.15.99:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99041379b0 con 0x7f990414be90 2024-03-25T18:36:55.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.621+0000 7f990ae2a640 1 librados: init done 2024-03-25T18:36:55.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.622+0000 7f990ae2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:55.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.622+0000 7f990ae2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:55.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.622+0000 7f990ae2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:55.625 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.622+0000 7f990ae2a640 1 -- 172.21.15.99:0/2754734893 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9904143740 con 0x7f990414be90 2024-03-25T18:36:55.649 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:11:56.242 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:55.661 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:11:56.220 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:55.949 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.948+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 <== mon.0 v1:172.21.15.99:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98fc001ca0 con 0x7f990414be90 2024-03-25T18:36:55.949 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.948+0000 7f99017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:55.949 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.948+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 <== mon.0 v1:172.21.15.99:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 737137807 0 0) 0x7f98fc003790 con 0x7f990414be90 2024-03-25T18:36:55.949 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.948+0000 7f99017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:55.949 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.948+0000 7f99017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:55.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.948+0000 7f99017fa640 10 monclient: dump: 2024-03-25T18:36:55.950 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2024-03-25T18:36:55.950 INFO:teuthology.misc.health.smithi099.stderr:fsid 71c41715-c00e-41a8-a30d-ef4898065160 2024-03-25T18:36:55.950 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:55.950 INFO:teuthology.misc.health.smithi099.stderr:created 2024-03-25T18:24:26.458245+0000 2024-03-25T18:36:55.950 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:55.950 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2024-03-25T18:36:55.950 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.a 2024-03-25T18:36:55.950 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.190:6789/0 mon.b 2024-03-25T18:36:55.950 INFO:teuthology.misc.health.smithi099.stderr:2: v1:172.21.15.99:6790/0 mon.c 2024-03-25T18:36:55.950 INFO:teuthology.misc.health.smithi099.stderr: 2024-03-25T18:36:55.950 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.948+0000 7f9902ffd640 4 set_mon_vals no callback set 2024-03-25T18:36:55.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.949+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 <== mon.0 v1:172.21.15.99:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3862107865 0 0) 0x7f98fc019760 con 0x7f990414be90 2024-03-25T18:36:55.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.949+0000 7f9908bc6640 10 monclient: get_auth_request con 0x7f98f006de10 auth_method 0 2024-03-25T18:36:55.951 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.950+0000 7f99017fa640 10 client.4737.objecter ms_handle_connect 0x7f98f006de10 2024-03-25T18:36:55.961 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.960+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 <== mon.0 v1:172.21.15.99:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2833420883 0 0) 0x7f98fc053410 con 0x7f990414be90 2024-03-25T18:36:55.961 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.960+0000 7f99017fa640 10 client.4737.objecter ms_dispatch 0x7f9904000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:55.962 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.960+0000 7f99017fa640 3 client.4737.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:55.962 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.960+0000 7f99017fa640 3 client.4737.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:55.962 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.960+0000 7f99017fa640 20 client.4737.objecter dump_active .. 0 homeless 2024-03-25T18:36:55.962 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.960+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 <== mon.0 v1:172.21.15.99:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98fc04ed50 con 0x7f990414be90 2024-03-25T18:36:55.962 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.960+0000 7f99017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:55.962 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:55.960+0000 7f99017fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:56.208 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.207+0000 7f990ae2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:56.209 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.207+0000 7f990ae2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:56.209 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.207+0000 7f990ae2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.99:6789/0 2024-03-25T18:36:56.209 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.207+0000 7f990ae2a640 1 -- 172.21.15.99:0/2754734893 --> v1:172.21.15.99:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99041401f0 con 0x7f990414be90 2024-03-25T18:36:56.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.230+0000 7f99017fa640 1 -- 172.21.15.99:0/2754734893 <== mon.0 v1:172.21.15.99:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 270404165) 0x7f98fc050e50 con 0x7f990414be90 2024-03-25T18:36:56.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.230+0000 7f99017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:56.231 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.230+0000 7f99017fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:56.231 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-25T18:36:56.231 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":11,"fsid":"71c41715-c00e-41a8-a30d-ef4898065160","created":"2024-03-25T18:25:15.610621+0000","modified":"2024-03-25T18:35:20.161948+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:35:20.161948+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b62a6f41-2c55-459e-b9f1-b90fc256eaea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f3b43bbc-3e21-4416-9c71-e9489aa1855a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"78a0d919-e9cb-4e29-8d0b-aceea8d3a59a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b1c2ae73-1010-4c54-93dc-dcac83badf88","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"031772d9-ed09-4828-87a7-6354210e6cc6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"553b866a-dbf1-48fd-8366-8c7522e3e20e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"da817edd-3a2f-4c44-b911-5d3fb2ebb686","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a443b4cf-d3f4-4411-8f9c-ee6f635d2504","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:56.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.232+0000 7f990ae2a640 10 librados: watch_flush enter 2024-03-25T18:36:56.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.232+0000 7f990ae2a640 10 librados: watch_flush exit 2024-03-25T18:36:56.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.232+0000 7f990ae2a640 20 client.4737.objecter shutdown clearing up homeless session... 2024-03-25T18:36:56.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.232+0000 7f990ae2a640 10 client.4737.objecter successfully canceled tick 2024-03-25T18:36:56.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.233+0000 7f990ae2a640 1 -- 172.21.15.99:0/2754734893 >> v1:172.21.15.190:6800/4220409699 conn(0x7f98f006de10 legacy=0x7f98f00702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:56.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.233+0000 7f990ae2a640 10 monclient: shutdown 2024-03-25T18:36:56.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.233+0000 7f990ae2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:56.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.233+0000 7f990ae2a640 1 -- 172.21.15.99:0/2754734893 >> v1:172.21.15.99:6789/0 conn(0x7f990414be90 legacy=0x7f99041370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:56.234 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.233+0000 7f990ae2a640 1 -- 172.21.15.99:0/2754734893 shutdown_connections 2024-03-25T18:36:56.235 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.233+0000 7f990ae2a640 1 -- 172.21.15.99:0/2754734893 >> 172.21.15.99:0/2754734893 conn(0x7f99040617a0 msgr2=0x7f99040b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:56.235 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.233+0000 7f990ae2a640 1 -- 172.21.15.99:0/2754734893 shutdown_connections 2024-03-25T18:36:56.235 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.233+0000 7f990ae2a640 1 -- 172.21.15.99:0/2754734893 wait complete. 2024-03-25T18:36:56.235 INFO:teuthology.misc.health.smithi099.stderr:2024-03-25T18:36:56.233+0000 7f990ae2a640 1 librados: shutdown 2024-03-25T18:36:56.245 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:56.245 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:36:56.247 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:36:56.247 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:36:56.247 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:36:58.310 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:11:36.314 35542== 2024-03-25T18:36:58.310 INFO:tasks.ceph.osd.0.smithi099.stderr:==00:00:11:36.314 35542== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:36:58.348 INFO:tasks.ceph.osd.0.smithi099.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:02.349 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:02.349 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:37:02.350 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:37:02.350 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:37:02.351 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:04.218 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:11:42.208 35545== 2024-03-25T18:37:04.218 INFO:tasks.ceph.osd.1.smithi099.stderr:==00:00:11:42.208 35545== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:04.256 INFO:tasks.ceph.osd.1.smithi099.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:08.452 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:37:08.452 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:37:08.453 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:37:08.453 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:10.288 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:11:48.279 35544== 2024-03-25T18:37:10.288 INFO:tasks.ceph.osd.2.smithi099.stderr:==00:00:11:48.279 35544== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:10.325 INFO:tasks.ceph.osd.2.smithi099.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:14.554 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:14.554 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:37:14.555 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:37:14.555 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:37:14.555 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:15.585 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:12:16.178 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:15.590 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:12:16.183 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:15.615 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:12:16.208 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:15.638 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:12:16.197 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:15.644 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:12:16.203 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:15.649 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:12:16.242 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:15.653 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:12:16.246 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:15.672 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:12:16.231 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:15.677 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:12:16.270 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:16.418 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:11:54.398 35549== 2024-03-25T18:37:16.418 INFO:tasks.ceph.osd.3.smithi099.stderr:==00:00:11:54.398 35549== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:16.463 INFO:tasks.ceph.osd.3.smithi099.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:20.657 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:37:20.657 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:37:20.657 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:37:20.657 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:22.562 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:12:00.529 35307== 2024-03-25T18:37:22.562 INFO:tasks.ceph.osd.4.smithi190.stderr:==00:00:12:00.529 35307== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:22.600 INFO:tasks.ceph.osd.4.smithi190.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:26.760 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:37:26.760 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:37:26.760 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:37:26.760 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:28.791 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:12:06.759 35305== 2024-03-25T18:37:28.791 INFO:tasks.ceph.osd.5.smithi190.stderr:==00:00:12:06.759 35305== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:28.828 INFO:tasks.ceph.osd.5.smithi190.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:32.862 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:32.863 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:37:32.863 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:37:32.863 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:37:32.863 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:34.674 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:12:12.641 35306== 2024-03-25T18:37:34.674 INFO:tasks.ceph.osd.6.smithi190.stderr:==00:00:12:12.641 35306== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:34.727 INFO:tasks.ceph.osd.6.smithi190.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:35.637 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:12:36.229 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.638 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:12:36.197 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.641 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:12:36.234 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.646 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:12:36.205 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.659 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:12:36.218 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.660 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:12:36.253 35231== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.692 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:12:36.285 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.694 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:12:36.287 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.711 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:12:36.304 35235== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:38.965 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:37:38.966 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:37:38.966 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:37:38.966 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:40.525 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:12:18.484 35308== 2024-03-25T18:37:40.526 INFO:tasks.ceph.osd.7.smithi190.stderr:==00:00:12:18.484 35308== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:40.556 INFO:tasks.ceph.osd.7.smithi190.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:45.068 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:45.069 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:37:45.069 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:37:45.069 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:37:45.070 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:37:45.070 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:45.095 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:37:45.095 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:37:45.095 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:45.096 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:37:45.096+0000 7f7617819640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34706) UID: 0 2024-03-25T18:37:45.096 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-03-25T18:37:45.096+0000 7f7617819640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:37:45.124 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:37:45.125 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:37:45.125 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:37:45.125 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:45.156 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-25T18:37:45.139+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35200) UID: 0 2024-03-25T18:37:45.157 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-25T18:37:45.141+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:37:45.594 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:12:46.187 35231== 2024-03-25T18:37:45.594 INFO:tasks.ceph.mon.a.smithi099.stderr:==00:00:12:46.187 35231== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:45.604 INFO:tasks.ceph.mon.a.smithi099.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:51.226 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:37:51.227 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:37:51.227 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:37:51.227 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:51.245 INFO:tasks.ceph.mon.c.smithi099.stderr:2024-03-25T18:37:51.231+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35202) UID: 0 2024-03-25T18:37:51.245 INFO:tasks.ceph.mon.c.smithi099.stderr:2024-03-25T18:37:51.232+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:39:58.005 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:14:58.597 35235== 2024-03-25T18:39:58.006 INFO:tasks.ceph.mon.c.smithi099.stderr:==00:00:14:58.597 35235== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:39:58.014 INFO:tasks.ceph.mon.c.smithi099.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:39:59.357 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:39:59.357 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:39:59.358 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:39:59.359 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:39:59.359 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:39:59.380 INFO:tasks.ceph.mon.b.smithi190.stderr:2024-03-25T18:39:59.363+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34704) UID: 0 2024-03-25T18:39:59.381 INFO:tasks.ceph.mon.b.smithi190.stderr:2024-03-25T18:39:59.364+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:39:59.875 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:15:00.434 34708== 2024-03-25T18:39:59.875 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:15:00.434 34708== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:39:59.884 INFO:tasks.ceph.mon.b.smithi190.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:40:05.461 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:40:05.461 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:40:05.462 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:40:05.462 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:40:05.463 DEBUG:teuthology.orchestra.run.smithi099:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:40:05.512 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-25T18:40:05.512 DEBUG:teuthology.orchestra.run.smithi099:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:40:05.616 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-25T18:40:05.616 DEBUG:teuthology.orchestra.run.smithi099:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:40:05.680 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-25T18:40:05.680 DEBUG:teuthology.orchestra.run.smithi099:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:40:05.738 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-25T18:40:05.738 DEBUG:teuthology.orchestra.run.smithi099:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:40:05.788 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-25T18:40:05.788 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:40:05.931 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-25T18:40:05.931 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:40:06.030 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-25T18:40:06.030 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:40:06.086 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi190.front.sepia.ceph.com 2024-03-25T18:40:06.086 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:40:06.145 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:40:06.147 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278/data/mon.a.tgz 2024-03-25T18:40:06.147 DEBUG:teuthology.orchestra.run.smithi099:> mktemp 2024-03-25T18:40:06.163 INFO:teuthology.orchestra.run.smithi099.stdout:/tmp/tmp.Vzj1Jf8ooi 2024-03-25T18:40:06.163 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Vzj1Jf8ooi 2024-03-25T18:40:06.260 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0666 /tmp/tmp.Vzj1Jf8ooi 2024-03-25T18:40:06.344 DEBUG:teuthology.orchestra.remote:smithi099:/tmp/tmp.Vzj1Jf8ooi is 132KB 2024-03-25T18:40:06.397 DEBUG:teuthology.orchestra.run.smithi099:> rm -fr /tmp/tmp.Vzj1Jf8ooi 2024-03-25T18:40:06.411 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278/data/mon.c.tgz 2024-03-25T18:40:06.411 DEBUG:teuthology.orchestra.run.smithi099:> mktemp 2024-03-25T18:40:06.465 INFO:teuthology.orchestra.run.smithi099.stdout:/tmp/tmp.UfnviYP7nt 2024-03-25T18:40:06.465 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.UfnviYP7nt 2024-03-25T18:40:06.553 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0666 /tmp/tmp.UfnviYP7nt 2024-03-25T18:40:06.637 DEBUG:teuthology.orchestra.remote:smithi099:/tmp/tmp.UfnviYP7nt is 133KB 2024-03-25T18:40:06.687 DEBUG:teuthology.orchestra.run.smithi099:> rm -fr /tmp/tmp.UfnviYP7nt 2024-03-25T18:40:06.701 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278/data/mon.b.tgz 2024-03-25T18:40:06.701 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-03-25T18:40:06.719 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.gZpLafSvPJ 2024-03-25T18:40:06.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.gZpLafSvPJ 2024-03-25T18:40:06.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.gZpLafSvPJ 2024-03-25T18:40:06.910 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.gZpLafSvPJ is 133KB 2024-03-25T18:40:06.962 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.gZpLafSvPJ 2024-03-25T18:40:06.977 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:40:06.977 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:40:06.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:40:07.361 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:40:07.604 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:40:07.606 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278/remote/smithi099/crash 2024-03-25T18:40:07.607 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:40:07.634 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278/remote/smithi190/crash 2024-03-25T18:40:07.635 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:40:07.662 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:40:07.662 DEBUG:teuthology.orchestra.run.smithi099:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:40:07.678 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:40:07.749 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:40:07.749 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34941.log 2024-03-25T18:40:07.749 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:40:07.749 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.34941.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:40:07.749 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:40:07.750 INFO:teuthology.orchestra.run.smithi190.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34941.log.gz 2024-03-25T18:40:07.750 INFO:teuthology.orchestra.run.smithi190.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:40:07.750 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:40:07.750 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:40:07.750 INFO:teuthology.orchestra.run.smithi190.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:40:07.750 INFO:teuthology.orchestra.run.smithi190.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:40:07.750 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34901.log 2024-03-25T18:40:07.750 INFO:teuthology.orchestra.run.smithi190.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:40:07.750 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35112.log 2024-03-25T18:40:07.750 INFO:teuthology.orchestra.run.smithi190.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:40:07.750 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.34901.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:40:07.750 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35112.log: gzip -5 --verbose 87.6% -- -- replaced with /var/log/ceph/ceph-client.admin.34901.log.gz /var/log/ceph/ceph-osd.4.log 2024-03-25T18:40:07.750 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:40:07.750 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35065.log 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35112.log.gz 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34861.log 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34981.log 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.34861.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.35065.log.gz 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34861.log.gz 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.34981.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34981.log.gz 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35021.log 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35021.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35152.log 2024-03-25T18:40:07.751 INFO:teuthology.orchestra.run.smithi190.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35021.log.gz 2024-03-25T18:40:07.752 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mon.b.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:40:07.752 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42335.log 2024-03-25T18:40:07.752 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41281.log 2024-03-25T18:40:07.752 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.42335.log: -- /var/log/ceph/ceph-client.admin.39705.log 2024-03-25T18:40:07.752 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41281.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:40:07.752 INFO:teuthology.orchestra.run.smithi099.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42335.log.gz 2024-03-25T18:40:07.752 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39471.log 2024-03-25T18:40:07.752 INFO:teuthology.orchestra.run.smithi099.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41281.log.gz 2024-03-25T18:40:07.752 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29390.log 2024-03-25T18:40:07.752 INFO:teuthology.orchestra.run.smithi099.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39705.log.gz 2024-03-25T18:40:07.752 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41710.log 2024-03-25T18:40:07.752 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph.tmp-client.admin.29390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39861.log 2024-03-25T18:40:07.752 INFO:teuthology.orchestra.run.smithi099.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39471.log.gz 2024-03-25T18:40:07.752 INFO:teuthology.orchestra.run.smithi099.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29390.log.gz 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39627.log 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41515.log 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41710.log.gz 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39627.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39861.log.gz 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5/var/log/ceph/ceph-client.admin.41515.log: --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39627.log.gz 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 86.6% 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41515.log.gz 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:40:07.753 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:40:07.754 INFO:teuthology.orchestra.run.smithi099.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:40:07.754 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37666.log 2024-03-25T18:40:07.754 INFO:teuthology.orchestra.run.smithi099.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:40:07.754 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39276.log 2024-03-25T18:40:07.754 INFO:teuthology.orchestra.run.smithi099.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:40:07.754 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.37666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41749.log 2024-03-25T18:40:07.754 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40538.log 2024-03-25T18:40:07.754 INFO:teuthology.orchestra.run.smithi099.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.37666.log.gz 2024-03-25T18:40:07.754 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41749.log: gzip 86.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.39276.log.gz 2024-03-25T18:40:07.754 INFO:teuthology.orchestra.run.smithi099.stderr: -- /var/log/ceph/ceph-client.admin.41983.log 2024-03-25T18:40:07.754 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42218.log 2024-03-25T18:40:07.754 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40538.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41749.log.gz 2024-03-25T18:40:07.754 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40852.log 2024-03-25T18:40:07.754 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.42218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40407.log 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41983.log.gz 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40538.log.gz 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41359.log 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42218.log.gz 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41398.log 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40852.log.gz 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41359.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40407.log.gz 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39900.log 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39744.log 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41359.log.gz 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42296.log 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41398.log.gz 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39744.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39900.log.gz 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39354.log 2024-03-25T18:40:07.755 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.42296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39549.log 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39744.log.gz 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40657.log 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42296.log.gz 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41671.log 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39354.log.gz 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38869.log 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39549.log.gz 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41671.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40657.log.gz 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35723.log 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.38869.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41671.log.gz 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40618.log 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38869.log.gz 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.35723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41086.log 2024-03-25T18:40:07.756 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42374.log 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35723.log.gz 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40618.log: /var/log/ceph/ceph-client.admin.41086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39118.log 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.42374.log: gzip -5 86.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40618.log.gz 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr: /var/log/ceph/ceph-client.admin.41476.log 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41086.log.gz 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41788.log 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42374.log.gz 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40696.log 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39118.log.gz 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40891.log 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41476.log.gz 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40696.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41788.log.gz 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39588.log 2024-03-25T18:40:07.757 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36590.log 2024-03-25T18:40:07.758 INFO:teuthology.orchestra.run.smithi099.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40696.log.gz 2024-03-25T18:40:07.758 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41164.log 2024-03-25T18:40:07.758 INFO:teuthology.orchestra.run.smithi099.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40891.log.gz 2024-03-25T18:40:07.758 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.36590.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39588.log.gz 2024-03-25T18:40:07.758 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40930.log 2024-03-25T18:40:07.758 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41866.log 2024-03-25T18:40:07.758 INFO:teuthology.orchestra.run.smithi099.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36590.log.gz 2024-03-25T18:40:07.758 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40930.log: 86.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.41164.log.gz 2024-03-25T18:40:07.758 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42139.log 2024-03-25T18:40:07.758 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41866.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:40:07.758 INFO:teuthology.orchestra.run.smithi099.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40930.log.gz 2024-03-25T18:40:07.758 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.42139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39510.log 2024-03-25T18:40:07.758 INFO:teuthology.orchestra.run.smithi099.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41866.log.gz 2024-03-25T18:40:07.758 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40173.log 2024-03-25T18:40:07.759 INFO:teuthology.orchestra.run.smithi099.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42139.log.gz 2024-03-25T18:40:07.759 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40969.log 2024-03-25T18:40:07.759 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41632.log 2024-03-25T18:40:07.759 INFO:teuthology.orchestra.run.smithi099.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39510.log.gz 2024-03-25T18:40:07.759 INFO:teuthology.orchestra.run.smithi099.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40173.log.gz 2024-03-25T18:40:07.759 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39079.log 2024-03-25T18:40:07.759 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40969.log: /var/log/ceph/ceph-client.admin.41632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39237.log 2024-03-25T18:40:07.759 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39079.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40969.log.gz 2024-03-25T18:40:07.759 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41593.log 2024-03-25T18:40:07.759 INFO:teuthology.orchestra.run.smithi099.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41632.log.gz 2024-03-25T18:40:07.759 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39237.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39079.log.gz 2024-03-25T18:40:07.759 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41242.log 2024-03-25T18:40:07.759 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39159.log 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41593.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39237.log.gz 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41944.log 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39159.log: gzip 87.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.41593.log.gz -- 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr: /var/log/ceph/ceph-client.admin.40251.log 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41242.log.gz 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41944.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39159.log.gzgzip 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41827.log 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40251.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41944.log.gz 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39822.log 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39783.log 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41827.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40251.log.gz 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35548.log 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39783.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41827.log.gz 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:40:07.760 INFO:teuthology.orchestra.run.smithi099.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39822.log.gz 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.35548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39978.log 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39783.log.gz 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-osd.3.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.35548.log.gz 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40056.log 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39315.log 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40446.log 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39978.log.gz 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42022.log 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40056.log.gz 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42100.log 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39315.log.gz 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.42022.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40446.log.gz 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40017.log 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.42100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41047.log 2024-03-25T18:40:07.761 INFO:teuthology.orchestra.run.smithi099.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42022.log.gz 2024-03-25T18:40:07.762 INFO:teuthology.orchestra.run.smithi099.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.42100.log.gz 2024-03-25T18:40:07.762 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35242.log 2024-03-25T18:40:07.762 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39939.log 2024-03-25T18:40:07.762 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41047.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40017.log.gz 2024-03-25T18:40:07.762 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.35242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40368.log 2024-03-25T18:40:07.762 INFO:teuthology.orchestra.run.smithi099.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41047.log.gz 2024-03-25T18:40:07.762 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40134.log 2024-03-25T18:40:07.762 INFO:teuthology.orchestra.run.smithi099.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.35242.log.gz 2024-03-25T18:40:07.762 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41905.log 2024-03-25T18:40:07.762 INFO:teuthology.orchestra.run.smithi099.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39939.log.gz 2024-03-25T18:40:07.762 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39393.log 2024-03-25T18:40:07.762 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40134.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40368.log.gz 2024-03-25T18:40:07.762 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41905.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:40:07.762 INFO:teuthology.orchestra.run.smithi099.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40134.log.gz 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39393.log: gzip 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41905.log.gz 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40774.log 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-mgr.y.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39393.log.gz 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35572.log 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39666.log 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40774.log.gz 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.35572.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5/var/log/ceph/ceph-client.admin.39666.log: --verbose -- /var/log/ceph/ceph-client.admin.42179.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35572.log.gz 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41437.log 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39666.log.gz 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.42179.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41437.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42179.log.gz 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40577.log 2024-03-25T18:40:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42061.log 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.40577.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41437.log.gz 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40577.log.gz 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40212.log 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.42061.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40212.log: 93.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42061.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41203.log 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40212.log.gz 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40329.log 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.41203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41320.log 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40329.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41203.log.gz 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40329.log.gz 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40813.log 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39432.log 2024-03-25T18:40:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40813.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41320.log.gz 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40813.log.gz 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40485.log 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39432.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39432.log.gz 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41125.log 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40290.log 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41125.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40485.log.gz 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41125.log.gz 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39198.log 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40735.log 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.39198.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40290.log.gz 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39198.log.gz 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41554.log 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40095.log 2024-03-25T18:40:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41008.log 2024-03-25T18:40:07.766 INFO:teuthology.orchestra.run.smithi099.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40735.log.gz 2024-03-25T18:40:07.766 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.40095.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41554.log.gzgzip 2024-03-25T18:40:07.766 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:40:07.766 INFO:teuthology.orchestra.run.smithi099.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40095.log.gz 2024-03-25T18:40:07.766 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.41008.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41008.log.gz 2024-03-25T18:40:07.766 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42257.log 2024-03-25T18:40:07.766 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.42257.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.42257.log.gz 2024-03-25T18:40:07.767 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35152.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35152.log.gz 2024-03-25T18:40:07.816 INFO:teuthology.orchestra.run.smithi190.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:40:08.548 INFO:teuthology.orchestra.run.smithi099.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:40:08.593 INFO:teuthology.orchestra.run.smithi190.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:40:08.633 INFO:teuthology.orchestra.run.smithi190.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:40:08.661 INFO:teuthology.orchestra.run.smithi190.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:40:08.816 INFO:teuthology.orchestra.run.smithi099.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:40:08.903 INFO:teuthology.orchestra.run.smithi099.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:40:08.928 INFO:teuthology.orchestra.run.smithi099.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:40:08.965 INFO:teuthology.orchestra.run.smithi190.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:40:12.748 INFO:teuthology.orchestra.run.smithi099.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:40:12.877 INFO:teuthology.orchestra.run.smithi099.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:40:12.879 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-25T18:40:12.879 INFO:teuthology.orchestra.run.smithi099.stderr:real 0m5.189s 2024-03-25T18:40:12.879 INFO:teuthology.orchestra.run.smithi099.stderr:user 0m14.067s 2024-03-25T18:40:12.879 INFO:teuthology.orchestra.run.smithi099.stderr:sys 0m0.531s 2024-03-25T18:40:12.998 INFO:teuthology.orchestra.run.smithi190.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:40:12.999 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-25T18:40:12.999 INFO:teuthology.orchestra.run.smithi190.stderr:real 0m5.277s 2024-03-25T18:40:13.000 INFO:teuthology.orchestra.run.smithi190.stderr:user 0m8.941s 2024-03-25T18:40:13.000 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m0.320s 2024-03-25T18:40:13.000 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:40:13.000 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278/remote/smithi099/log 2024-03-25T18:40:13.001 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:40:15.403 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278/remote/smithi190/log 2024-03-25T18:40:15.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:40:16.957 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:40:17.172 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5777095dd1f54c70aa4aadf14b16c70c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:40:17.176 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:40:17.186 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:40:17.231 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:40:17.231 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:40:17.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:40:17.323 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:40:17.332 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:40:17.332 DEBUG:teuthology.orchestra.run.smithi099:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:40:17.335 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:40:17.348 INFO:teuthology.orchestra.run.smithi099.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:40:17.351 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:40:17.353 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:40:17.353 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-03-25T18:40:17.353 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 18 +471us[ +486us] +/- 78ms 2024-03-25T18:40:17.353 INFO:teuthology.orchestra.run.smithi099.stdout:^* hv02.front.sepia.ceph.com 3 6 377 16 -1057us[-1042us] +/- 63ms 2024-03-25T18:40:17.354 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +763us[ +778us] +/- 68ms 2024-03-25T18:40:17.354 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:40:17.355 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:40:17.355 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-03-25T18:40:17.355 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 10 +486us[ +486us] +/- 78ms 2024-03-25T18:40:17.355 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv02.front.sepia.ceph.com 3 6 377 14 -1053us[-1048us] +/- 63ms 2024-03-25T18:40:17.356 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 +770us[ +775us] +/- 68ms 2024-03-25T18:40:17.356 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:40:17.357 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:40:17.444 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:40:17.445 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:40:17.480 DEBUG:teuthology.orchestra.run.smithi099:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:40:17.553 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:40:17.625 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:40:17.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:40:17.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:40:17.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:40:17.690 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:40:17.700 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:40:17.745 INFO:teuthology.task.internal:Duration was 7908.789984 seconds 2024-03-25T18:40:17.746 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:40:17.805 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:40:17.805 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:40:17.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:40:17.851 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:40:17.853 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:40:18.173 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:40:18.173 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi099.front.sepia.ceph.com 2024-03-25T18:40:18.174 DEBUG:teuthology.orchestra.run.smithi099:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:40:18.195 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-03-25T18:40:18.195 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:40:18.219 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:40:18.219 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:40:18.238 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:40:18.288 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:40:18.289 DEBUG:teuthology.orchestra.run.smithi099:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:40:18.309 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:40:18.431 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:40:18.442 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:40:18.442 DEBUG:teuthology.orchestra.run.smithi099:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:40:18.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:40:18.496 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:40:18.506 DEBUG:teuthology.orchestra.run.smithi099:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:40:18.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:40:18.535 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = core 2024-03-25T18:40:18.561 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-03-25T18:40:18.587 DEBUG:teuthology.orchestra.run.smithi099:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:40:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:40:18.621 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:40:18.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:40:18.641 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:40:18.651 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:40:18.652 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278/remote/smithi099 2024-03-25T18:40:18.652 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:40:18.697 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278/remote/smithi190 2024-03-25T18:40:18.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:40:18.732 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:40:18.732 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:40:18.735 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:40:18.782 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:40:18.793 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:40:18.793 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:40:18.813 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:40:18.813 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:40:18.816 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:40:18.827 INFO:teuthology.orchestra.run.smithi099.stdout: 265819 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:40 /home/ubuntu/cephtest 2024-03-25T18:40:18.836 INFO:teuthology.orchestra.run.smithi190.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:40 /home/ubuntu/cephtest 2024-03-25T18:40:18.838 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:40:18.859 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:40:18.957 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:40:18.984 DEBUG:teuthology.nuke:shortname: smithi099 2024-03-25T18:40:18.984 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:40:19.010 DEBUG:teuthology.nuke:shortname: smithi190 2024-03-25T18:40:19.010 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:40:19.014 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:18:06.649998', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGtxbdLy2UHaHA5gRIf3MR5NU+UzyKuyLZnQy02lhjxuvfdtMPt2OxqNQS24ovTq2LYanLIoZBf0ViH2OnEuAKo='} 2024-03-25T18:40:19.037 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621278', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:18:06.648321', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCYd91K16MqN2SbtpXlvonTKb/ApH9+7u27YGHa817euhxeRbawaNy+ucBzzgJtxkgmTJjO1E+7tv7Dyw3xr/rU='} 2024-03-25T18:40:19.060 INFO:teuthology.orchestra.console.smithi099:Power off 2024-03-25T18:40:19.061 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:40:19.085 DEBUG:teuthology.orchestra.console.smithi099:power off output: Chassis Power Control: Down/Off 2024-03-25T18:40:19.085 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:40:19.103 INFO:teuthology.orchestra.console.smithi190:Power off 2024-03-25T18:40:19.104 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:40:19.114 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-03-25T18:40:19.129 DEBUG:teuthology.orchestra.console.smithi190:power off output: Chassis Power Control: Down/Off 2024-03-25T18:40:19.129 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:40:19.153 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-03-25T18:40:23.116 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:40:23.155 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:40:23.226 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-03-25T18:40:23.265 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-03-25T18:40:27.230 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:40:27.266 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:40:27.341 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-03-25T18:40:27.376 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is off 2024-03-25T18:40:27.476 INFO:teuthology.orchestra.console.smithi190:Power off completed 2024-03-25T18:40:31.342 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:40:31.456 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is off 2024-03-25T18:40:31.557 INFO:teuthology.orchestra.console.smithi099:Power off completed 2024-03-25T18:40:31.662 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7908.789983987808 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5777095dd1f54c70aa4aadf14b16c70c status: fail success: false 2024-03-25T18:40:31.663 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:40:31.752 INFO:teuthology.run:FAIL